Sep 15 21:26:02.061085 ip-10-0-135-148 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 21:26:02.506836 ip-10-0-135-148 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:26:02.506836 ip-10-0-135-148 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 21:26:02.506836 ip-10-0-135-148 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:26:02.506836 ip-10-0-135-148 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 21:26:02.506836 ip-10-0-135-148 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 21:26:02.508098 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.508004    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 21:26:02.513702 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513683    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:26:02.513702 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513699    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:26:02.513702 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513704    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:26:02.513702 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513708    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513715    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513719    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513723    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513728    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513733    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513737    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513741    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513745    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513749    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513769    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513773    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513777    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513781    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513785    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513789    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513794    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513798    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513801    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513805    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:26:02.513950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513809    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513813    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513817    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513824    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513830    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513834    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513839    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513843    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513848    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513852    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513856    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513860    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513864    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513869    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513876    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513881    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513885    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513889    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513893    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513898    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:26:02.514743 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513902    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513906    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513910    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513914    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513919    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513923    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513927    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513933    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513937    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513942    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513946    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513949    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513953    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513958    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513962    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513966    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513970    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513974    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513978    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513982    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:26:02.515434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513986    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513993    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.513999    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514004    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514009    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514014    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514021    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514026    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514031    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514036    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514044    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514048    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514052    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514057    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514062    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514067    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514071    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514075    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514080    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:26:02.515946 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514085    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514089    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514093    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514097    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514101    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514105    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514109    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514114    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514118    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514122    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514126    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514130    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514134    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514140    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514146    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514151    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.514156    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515328    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515341    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:26:02.516418 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515345    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515348    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515351    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515354    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515358    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515360    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515363    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515365    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515368    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515371    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515374    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515377    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515380    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515382    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515385    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515388    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515390    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515393    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515395    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515397    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:26:02.516913 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515400    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515402    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515405    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515407    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515410    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515412    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515416    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515421    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515424    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515427    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515430    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515435    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515438    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515441    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515444    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515446    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515449    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515452    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515454    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515456    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:26:02.517657 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515459    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515462    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515464    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515467    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515469    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515471    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515474    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515477    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515479    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515482    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515484    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515487    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515490    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515493    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515495    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515498    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515500    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515503    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515505    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515508    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:26:02.518422 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515510    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515513    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515516    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515518    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515521    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515524    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515527    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515529    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515531    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515534    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515537    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515541    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515544    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515546    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515549    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515551    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515555    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515558    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515560    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:26:02.518950 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515563    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515566    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515569    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515571    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515573    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515576    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515578    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515581    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515583    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515585    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515588    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515591    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515594    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515596    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515599    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515602    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515605    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.515607    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516004    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:26:02.519421 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516010    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516013    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516015    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516018    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516021    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516024    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516027    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516029    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516033    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516037    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516041    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516043    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516046    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516049    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516052    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516055    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516057    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516060    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516063    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516065    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:26:02.519918 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516068    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516070    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516073    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516075    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516077    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516080    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516082    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516089    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516092    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516095    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516098    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516101    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516103    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516106    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516108    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516111    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516113    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516116    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516118    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:26:02.520409 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516121    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516124    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516126    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516129    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516132    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516135    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516137    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516140    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516142    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516145    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516147    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516150    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516152    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516154    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516157    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516159    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516162    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516166    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516169    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:26:02.520979 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516172    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516176    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516180    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516183    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516185    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516188    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516191    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516193    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516196    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516198    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516201    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516203    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516206    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516208    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516210    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516213    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516215    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516218    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516220    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:26:02.521455 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516224    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516226    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516228    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516231    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516233    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516236    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516239    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516241    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516243    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516246    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516249    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516251    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516255    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516257    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516261    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516264    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516267    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516270    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516274    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516277    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:26:02.521937 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.516280    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516360    2575 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516394    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516400    2575 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516405    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516409    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516412    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516416    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516421    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516424    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516427    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516431    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516434    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516437    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516440    2575 flags.go:64] FLAG: --cgroup-root=""
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516443    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516446    2575 flags.go:64] FLAG: --client-ca-file=""
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516448    2575 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516451    2575 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516456    2575 flags.go:64] FLAG: --cluster-domain=""
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516458    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516461    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516464    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516467    2575 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 21:26:02.522440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516471    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516474    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516478    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516482    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516485    2575 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516488    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516491    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516494    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516496    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516501    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516504    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516506    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516509    2575 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516512    2575 flags.go:64] FLAG: --enable-server="true"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516515    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516519    2575 flags.go:64] FLAG: --event-burst="100"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516522    2575 flags.go:64] FLAG: --event-qps="50"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516525    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516527    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516530    2575 flags.go:64] FLAG: --eviction-hard=""
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516534    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516537    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516539    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516542    2575 flags.go:64] FLAG: --eviction-soft=""
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516545    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 21:26:02.523040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516548    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516550    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516553    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516556    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516559    2575 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516561    2575 flags.go:64] FLAG: --feature-gates=""
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516565    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516568    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516571    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516575    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516578    2575 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516581    2575 flags.go:64] FLAG: --help="false"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516584    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-135-148.ec2.internal"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516587    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516590    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516593    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516596    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516600    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516603    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516606    2575 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516608    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516611    2575 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516614    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516617    2575 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 21:26:02.523661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516620    2575 flags.go:64] FLAG: --kube-reserved=""
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516623    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516625    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516628    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516631    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516634    2575 flags.go:64] FLAG: --lock-file=""
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516636    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516639    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516642    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516647    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516650    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516653    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516655    2575 flags.go:64] FLAG: --logging-format="text"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516658    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516662    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516664    2575 flags.go:64] FLAG: --manifest-url=""
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516667    2575 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516672    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516675    2575 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516679    2575 flags.go:64] FLAG: --max-pods="110"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516682    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516685    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516687    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516690    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516693    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 21:26:02.524320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516696    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516700    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516707    2575 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516710    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516714    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516717    2575 flags.go:64] FLAG: --pod-cidr=""
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516719    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516725    2575 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516728    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516731    2575 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516734    2575 flags.go:64] FLAG: --port="10250"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516737    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516740    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-067215f6722367743"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516743    2575 flags.go:64] FLAG: --qos-reserved=""
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516745    2575 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516748    2575 flags.go:64] FLAG: --register-node="true"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516765    2575 flags.go:64] FLAG: --register-with-taints=""
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516776    2575 flags.go:64] FLAG: --registry-burst="10"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516780    2575 flags.go:64] FLAG: --registry-qps="5"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516783    2575 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516786    2575 flags.go:64] FLAG: --reserved-memory=""
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516789    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516792    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516795    2575 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516804    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 21:26:02.524969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516807    2575 flags.go:64] FLAG: --runonce="false"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516810    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516813    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516816    2575 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516819    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516822    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516825    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516828    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516830    2575 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516834    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516837    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516840    2575 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516843    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516845    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516848    2575 flags.go:64] FLAG: --system-cgroups=""
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516851    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516856    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516859    2575 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516862    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516866    2575 flags.go:64] FLAG: --tls-min-version=""
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516869    2575 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516872    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516874    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516878    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516880    2575 flags.go:64] FLAG: --v="2"
Sep 15 21:26:02.525679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516884    2575 flags.go:64] FLAG: --version="false"
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516888    2575 flags.go:64] FLAG: --vmodule=""
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516893    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.516896    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517654    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517666    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517671    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517678    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517683    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517687    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517691    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517695    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517701    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517710    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517714    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517718    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517722    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517737    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517742    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517746    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:26:02.526308 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517768    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517773    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517777    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517781    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517786    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517790    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517799    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517803    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517807    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517811    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517815    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517819    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517824    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517827    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517833    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517837    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517841    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517848    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517853    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:26:02.526822 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517864    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517868    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517873    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517877    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517881    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517886    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517890    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517895    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517900    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517904    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517908    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517914    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517923    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517928    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517932    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517937    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517941    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517945    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517950    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517954    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:26:02.527281 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517958    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517962    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517966    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517970    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517979    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517983    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517987    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517993    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.517996    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518001    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518005    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518009    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518014    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518018    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518023    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518027    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518031    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518041    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518045    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:26:02.527793 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518052    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518057    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518061    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518065    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518069    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518074    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518081    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518086    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518091    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518096    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518106    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518110    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518114    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518118    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518123    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518127    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518131    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518135    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518139    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518143    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:26:02.528275 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518150    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:26:02.528749 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518154    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:26:02.528749 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518163    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:26:02.528749 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518167    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:26:02.528749 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.518172    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:26:02.528749 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.518776    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:26:02.528749 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.526129    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 21:26:02.528749 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.526145    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 21:26:02.528749 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.526773    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 21:26:02.528749 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.526781    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 21:26:02.528749 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526838    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:26:02.528749 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526844    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:26:02.528749 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526848    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:26:02.528749 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526851    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:26:02.528749 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526854    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:26:02.528749 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526857    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526859    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526863    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526865    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526868    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526871    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526874    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526876    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526879    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526882    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526884    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526887    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526890    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526893    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526895    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526898    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526900    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526903    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526905    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526908    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:26:02.529155 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526911    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526914    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526916    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526919    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526922    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526925    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526928    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526930    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526933    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526935    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526938    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526940    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526943    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526946    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526948    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526951    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526954    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526956    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526958    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526961    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:26:02.529639 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526965    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526968    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526971    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526974    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526977    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526981    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526984    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526988    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526990    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526993    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526996    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.526999    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527001    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527004    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527006    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527009    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527013    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527016    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527018    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527021    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:26:02.530145 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527023    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527026    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527028    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527031    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527034    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527037    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527039    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527042    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527044    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527047    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527050    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527052    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527055    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527057    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527060    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527063    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527065    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527068    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527070    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527072    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:26:02.530633 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527075    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:26:02.531128 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527077    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:26:02.531128 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527080    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:26:02.531128 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527083    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:26:02.531128 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527087    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:26:02.531128 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527091    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:26:02.531128 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527093    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:26:02.531128 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527097    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:26:02.531128 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527100    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:26:02.531128 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527103    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:26:02.531128 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527105    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:26:02.531128 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527108    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:26:02.531128 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527110    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:26:02.531128 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527113    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:26:02.531128 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.527118    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:26:02.531128 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527237    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527241    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527244    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527247    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527250    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527253    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527255    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527258    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527262    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527265    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527269    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527272    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527275    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527277    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527280    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527283    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527287    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527290    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 21:26:02.531494 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527293    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527296    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527299    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527301    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527304    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527306    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527308    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527311    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527314    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527317    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527320    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527322    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527325    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527327    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527330    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527332    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527335    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527338    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527340    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527342    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 21:26:02.531951 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527345    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527347    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527350    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527352    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527354    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527357    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527360    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527362    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527365    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527367    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527370    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527372    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527374    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527378    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527381    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527383    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527386    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527388    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527390    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527393    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 21:26:02.532442 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527396    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527399    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527401    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527404    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527406    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527409    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527411    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527414    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527416    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527419    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527421    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527423    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527426    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527428    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527431    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527433    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527435    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527438    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527441    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 21:26:02.533054 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527444    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527447    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527449    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527451    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527454    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527456    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527458    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527461    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527463    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527466    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527468    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527471    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527474    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527477    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527479    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527482    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527484    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527487    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527489    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527492    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 21:26:02.533539 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527494    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 21:26:02.534042 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:02.527497    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 21:26:02.534042 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.527501    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 21:26:02.534042 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.527616    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 21:26:02.534042 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.529847    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 21:26:02.534042 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.531298    2575 server.go:1019] "Starting client certificate rotation"
Sep 15 21:26:02.534042 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.531405    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 21:26:02.534042 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.531437    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 21:26:02.534042 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.533906    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 21:26:02.537998 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.537985    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 21:26:02.550818 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.550795    2575 log.go:25] "Validated CRI v1 runtime API"
Sep 15 21:26:02.554979 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.554964    2575 log.go:25] "Validated CRI v1 image API"
Sep 15 21:26:02.555862 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.555843    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 21:26:02.560217 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.560192    2575 fs.go:135] Filesystem UUIDs: map[3e187090-e1ae-4202-bd14-420a082899ad:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 af9092ff-d013-423d-b361-1eb2ab481494:/dev/nvme0n1p3]
Sep 15 21:26:02.560274 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.560217    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 21:26:02.562858 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.562831    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 21:26:02.566411 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.566303    2575 manager.go:216] Machine: {Timestamp:2026-09-15 21:26:02.564831825 +0000 UTC m=+0.376840259 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3105049 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2350da37a382e23c9fe6af152d91fd SystemUUID:ec2350da-37a3-82e2-3c9f-e6af152d91fd BootID:2a4abb0e-1351-4126-abe9-f3f09aa9a115 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:db:60:c8:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:db:60:c8:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:41:6a:95:88:e3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 21:26:02.566411 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.566404    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 21:26:02.566527 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.566485    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 21:26:02.569397 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.569375    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 21:26:02.569527 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.569400    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-148.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 21:26:02.570259 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.570248    2575 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 21:26:02.570290 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.570261    2575 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 21:26:02.570290 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.570269    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 21:26:02.570290 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.570284    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 21:26:02.571017 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.571007    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 21:26:02.574903 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.574892    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 21:26:02.575015 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.575005    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 21:26:02.575051 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.575034    2575 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 21:26:02.575051 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.575044    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 21:26:02.575108 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.575054    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 21:26:02.575108 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.575063    2575 kubelet.go:398] "Adding apiserver pod source"
Sep 15 21:26:02.575108 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.575072    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 21:26:02.575515 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.575502    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 21:26:02.575569 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.575527    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 21:26:02.577791 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.577772    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 21:26:02.580158 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.580144    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 21:26:02.580215 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.580174    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 21:26:02.581787 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.581773    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 21:26:02.581787 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.581789    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 21:26:02.581916 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.581795    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 21:26:02.581916 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.581801    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 21:26:02.581916 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.581809    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 21:26:02.581916 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.581814    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 21:26:02.581916 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.581827    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 21:26:02.581916 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.581832    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 21:26:02.581916 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.581838    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 21:26:02.581916 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.581844    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 21:26:02.581916 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.581852    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 21:26:02.581916 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.581857    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 21:26:02.582656 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.582645    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 21:26:02.582656 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.582654    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 21:26:02.585695 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.585674    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jsvp6"
Sep 15 21:26:02.586746 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.586731    2575 server.go:1268] "Started kubelet"
Sep 15 21:26:02.586878 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.586857    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 21:26:02.586938 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.586905    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 21:26:02.586991 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.586902    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 21:26:02.586991 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.586978    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-148.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 21:26:02.587086 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.586965    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 21:26:02.587086 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.587020    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 21:26:02.587600 ip-10-0-135-148 systemd[1]: Started Kubernetes Kubelet.
Sep 15 21:26:02.588516 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.588183    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 21:26:02.589106 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.589094    2575 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 21:26:02.591940 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.591922    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 21:26:02.593137 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.593121    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 21:26:02.593251 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.593231    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jsvp6"
Sep 15 21:26:02.593806 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.593791    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 21:26:02.593889 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.593873    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found"
Sep 15 21:26:02.594267 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.593131    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-148.ec2.internal.18d59ba59edb129b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-148.ec2.internal,UID:ip-10-0-135-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-148.ec2.internal,},FirstTimestamp:2026-09-15 21:26:02.586706587 +0000 UTC m=+0.398715024,LastTimestamp:2026-09-15 21:26:02.586706587 +0000 UTC m=+0.398715024,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-148.ec2.internal,}"
Sep 15 21:26:02.594698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.594684    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 21:26:02.594698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.594698    2575 factory.go:55] Registering systemd factory
Sep 15 21:26:02.594827 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.594704    2575 factory.go:223] Registration of the systemd container factory successfully
Sep 15 21:26:02.594827 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.594773    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 21:26:02.594827 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.594784    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 21:26:02.594992 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.594878    2575 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 21:26:02.594992 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.594885    2575 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 21:26:02.594992 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.594913    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 21:26:02.594992 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.594929    2575 factory.go:153] Registering CRI-O factory
Sep 15 21:26:02.594992 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.594947    2575 factory.go:223] Registration of the crio container factory successfully
Sep 15 21:26:02.594992 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.594964    2575 factory.go:103] Registering Raw factory
Sep 15 21:26:02.595254 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.595003    2575 manager.go:1196] Started watching for new ooms in manager
Sep 15 21:26:02.595383 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.595371    2575 manager.go:349] Starting recovery of all containers
Sep 15 21:26:02.601161 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.601132    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 21:26:02.601265 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.601209    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 21:26:02.607336 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.607314    2575 manager.go:354] Recovery completed
Sep 15 21:26:02.611738 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.611723    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:26:02.613641 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.613625    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:26:02.613719 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.613661    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:26:02.613719 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.613676    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:26:02.614339 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.614324    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 21:26:02.614339 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.614336    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 21:26:02.614475 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.614342    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:26:02.614475 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.614351    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 21:26:02.615097 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.615084    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:26:02.615148 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.615109    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:26:02.615148 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.615118    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:26:02.616302 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.616292    2575 policy_none.go:49] "None policy: Start"
Sep 15 21:26:02.616335 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.616308    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 21:26:02.616335 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.616318    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 21:26:02.618199 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.618186    2575 policy_none.go:47] "Start"
Sep 15 21:26:02.655505 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.655488    2575 manager.go:337] "Starting Device Plugin manager"
Sep 15 21:26:02.677812 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.655544    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 21:26:02.677812 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.655560    2575 server.go:87] "Starting device plugin registration server"
Sep 15 21:26:02.677812 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.656045    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 21:26:02.677812 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.656064    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 21:26:02.677812 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.656223    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 21:26:02.677812 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.656302    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 21:26:02.677812 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.656311    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 21:26:02.677812 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.656785    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 21:26:02.677812 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.656825    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-148.ec2.internal\" not found"
Sep 15 21:26:02.739072 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.739035    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 21:26:02.740434 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.740416    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 21:26:02.740492 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.740483    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 21:26:02.740525 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.740510    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 21:26:02.740575 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.740560    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 21:26:02.743376 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.743352    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:26:02.756541 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.756456    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:26:02.758533 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.758512    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:26:02.758624 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.758542    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:26:02.758624 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.758559    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:26:02.758624 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.758581    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-148.ec2.internal"
Sep 15 21:26:02.766328 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.766309    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-148.ec2.internal"
Sep 15 21:26:02.766386 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.766329    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-148.ec2.internal\": node \"ip-10-0-135-148.ec2.internal\" not found"
Sep 15 21:26:02.783253 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.783231    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found"
Sep 15 21:26:02.840934 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.840906    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal"]
Sep 15 21:26:02.841024 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.840993    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:26:02.843068 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.843053    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:26:02.843144 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.843083    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:26:02.843144 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.843106    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:26:02.845169 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.845155    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:26:02.845298 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.845285    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal"
Sep 15 21:26:02.845334 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.845312    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:26:02.845985 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.845971    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:26:02.845985 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.845981    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:26:02.846073 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.845999    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:26:02.846073 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.846012    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:26:02.846073 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.846001    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:26:02.846158 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.846103    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:26:02.848062 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.848047    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal"
Sep 15 21:26:02.848142 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.848071    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 21:26:02.849175 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.849162    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 21:26:02.849257 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.849182    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 21:26:02.849257 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.849192    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeHasSufficientPID"
Sep 15 21:26:02.862298 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.862278    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-148.ec2.internal\" not found" node="ip-10-0-135-148.ec2.internal"
Sep 15 21:26:02.866016 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.866002    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-148.ec2.internal\" not found" node="ip-10-0-135-148.ec2.internal"
Sep 15 21:26:02.883571 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.883553    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found"
Sep 15 21:26:02.896506 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.896484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2362bd6974628566f33836616227d001-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"2362bd6974628566f33836616227d001\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal"
Sep 15 21:26:02.896595 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.896528    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2362bd6974628566f33836616227d001-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"2362bd6974628566f33836616227d001\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal"
Sep 15 21:26:02.896595 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.896546    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea4e7f0d7b04a6dd59e41a5028a23abd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-148.ec2.internal\" (UID: \"ea4e7f0d7b04a6dd59e41a5028a23abd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal"
Sep 15 21:26:02.983694 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:02.983654    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found"
Sep 15 21:26:02.997002 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.996979    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2362bd6974628566f33836616227d001-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"2362bd6974628566f33836616227d001\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal"
Sep 15 21:26:02.997052 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.997015    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2362bd6974628566f33836616227d001-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"2362bd6974628566f33836616227d001\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal"
Sep 15 21:26:02.997052 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.997032    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea4e7f0d7b04a6dd59e41a5028a23abd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-148.ec2.internal\" (UID: \"ea4e7f0d7b04a6dd59e41a5028a23abd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal"
Sep 15 21:26:02.997135 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.997059    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ea4e7f0d7b04a6dd59e41a5028a23abd-config\") pod \"kube-apiserver-proxy-ip-10-0-135-148.ec2.internal\" (UID: \"ea4e7f0d7b04a6dd59e41a5028a23abd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal"
Sep 15 21:26:02.997135 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.997083    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2362bd6974628566f33836616227d001-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"2362bd6974628566f33836616227d001\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal"
Sep 15 21:26:02.997135 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:02.997099    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2362bd6974628566f33836616227d001-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal\" (UID: \"2362bd6974628566f33836616227d001\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal"
Sep 15 21:26:03.084488 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:03.084422    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found"
Sep 15 21:26:03.164972 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:03.164947    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal"
Sep 15 21:26:03.168203 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:03.168152    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal"
Sep 15 21:26:03.184570 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:03.184547    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found"
Sep 15 21:26:03.284688 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:03.284636    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found"
Sep 15 21:26:03.385282 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:03.385202    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found"
Sep 15 21:26:03.443447 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:03.443420    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:26:03.485333 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:03.485302    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found"
Sep 15 21:26:03.530789 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:03.530749    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 21:26:03.531182 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:03.530910    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 21:26:03.531182 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:03.530934    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 21:26:03.586233 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:03.586189    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found"
Sep 15 21:26:03.599262 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:03.599231    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 21:21:03 +0000 UTC" deadline="2028-03-20 09:42:44.972773647 +0000 UTC"
Sep 15 21:26:03.599262 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:03.599258    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13236h16m41.373518784s"
Sep 15 21:26:03.686887 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:03.686855    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found"
Sep 15 21:26:03.762814 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:03.762776    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2362bd6974628566f33836616227d001.slice/crio-dbea59e7dcd04babb86b450fca43e1d2d7b9de56c00e052486b9f8358bf2a5e0 WatchSource:0}: Error finding container dbea59e7dcd04babb86b450fca43e1d2d7b9de56c00e052486b9f8358bf2a5e0: Status 404 returned error can't find the container with id dbea59e7dcd04babb86b450fca43e1d2d7b9de56c00e052486b9f8358bf2a5e0
Sep 15 21:26:03.767235 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:03.767221    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:26:03.787394 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:03.787372    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-148.ec2.internal\" not found"
Sep 15 21:26:03.788499 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:03.788480    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:26:03.792878 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:03.792855    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4e7f0d7b04a6dd59e41a5028a23abd.slice/crio-74f9f9a042ddead316735970633a125eea44550bdbed44466c59486e362cdd69 WatchSource:0}: Error finding container 74f9f9a042ddead316735970633a125eea44550bdbed44466c59486e362cdd69: Status 404 returned error can't find the container with id 74f9f9a042ddead316735970633a125eea44550bdbed44466c59486e362cdd69
Sep 15 21:26:03.793980 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:03.793966    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal"
Sep 15 21:26:03.807165 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:03.807138    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 21:26:03.808426 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:03.808413    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal"
Sep 15 21:26:03.819829 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:03.819804    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 21:26:04.016692 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.016588    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:26:04.528593 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.528554    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:26:04.576187 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.576155    2575 apiserver.go:52] "Watching apiserver"
Sep 15 21:26:04.584166 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.584137    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 21:26:04.584571 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.584549    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf","openshift-cluster-node-tuning-operator/tuned-lcz99","openshift-network-diagnostics/network-check-target-rsbdn","openshift-ovn-kubernetes/ovnkube-node-mpwmh","kube-system/global-pull-secret-syncer-v6j25","openshift-dns/node-resolver-rwmx9","openshift-image-registry/node-ca-wwhph","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal","openshift-multus/multus-additional-cni-plugins-jmmgp","openshift-multus/multus-js9vq","openshift-multus/network-metrics-daemon-b46fs","openshift-network-operator/iptables-alerter-tjsqw","kube-system/konnectivity-agent-zvtxn","kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal"]
Sep 15 21:26:04.587327 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.587305    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wwhph"
Sep 15 21:26:04.589856 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.589792    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 21:26:04.589856 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.589822    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 21:26:04.589856 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.589849    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-h5x6x\""
Sep 15 21:26:04.590068 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.589878    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zvtxn"
Sep 15 21:26:04.590068 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.589837    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 21:26:04.591966 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.591946    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 21:26:04.592290 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.592268    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 21:26:04.592408 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.592387    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tm8hk\""
Sep 15 21:26:04.592574 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.592556    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:04.592663 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:04.592640    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:04.593476 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.593455    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 21:26:04.594562 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.594543    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.598122 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.598096    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 21:26:04.598246 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.598225    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 21:26:04.598350 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.598333    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 21:26:04.598521 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.598497    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-25bms\""
Sep 15 21:26:04.598625 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.598505    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:04.598722 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:04.598697    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:04.599869 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.599048    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 21:26:04.599869 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.599228    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 21:26:04.599869 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.599366    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 21:26:04.600542 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.600523    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rwmx9"
Sep 15 21:26:04.602553 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.602536    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7rgs9\""
Sep 15 21:26:04.602635 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.602570    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 21:26:04.602997 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.602974    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 21:26:04.603116 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.603098    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.604783 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.604763    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/722562f8-7296-47bd-8fec-1d6791c5afd0-konnectivity-ca\") pod \"konnectivity-agent-zvtxn\" (UID: \"722562f8-7296-47bd-8fec-1d6791c5afd0\") " pod="kube-system/konnectivity-agent-zvtxn"
Sep 15 21:26:04.604930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.604800    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-run-systemd\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.604930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.604914    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-run-openvswitch\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605076 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.604944    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pzvz\" (UniqueName: \"kubernetes.io/projected/4eb4bf49-c243-426b-a143-22c987695db4-kube-api-access-5pzvz\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605076 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.604972    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21784d6c-c661-42eb-8b09-41376dacc1a3-dbus\") pod \"global-pull-secret-syncer-v6j25\" (UID: \"21784d6c-c661-42eb-8b09-41376dacc1a3\") " pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:04.605076 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.604999    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-slash\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605076 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605023    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-etc-openvswitch\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605076 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605049    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-run-ovn\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605326 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605077    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605326 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605138    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxwnx\" (UniqueName: \"kubernetes.io/projected/6b3f47d0-ec5b-4699-91a9-160ea99f70fc-kube-api-access-bxwnx\") pod \"node-ca-wwhph\" (UID: \"6b3f47d0-ec5b-4699-91a9-160ea99f70fc\") " pod="openshift-image-registry/node-ca-wwhph"
Sep 15 21:26:04.605326 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605182    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-kubelet\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605326 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605209    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-cni-bin\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605326 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605261    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21784d6c-c661-42eb-8b09-41376dacc1a3-kubelet-config\") pod \"global-pull-secret-syncer-v6j25\" (UID: \"21784d6c-c661-42eb-8b09-41376dacc1a3\") " pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:04.605326 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605291    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6b3f47d0-ec5b-4699-91a9-160ea99f70fc-serviceca\") pod \"node-ca-wwhph\" (UID: \"6b3f47d0-ec5b-4699-91a9-160ea99f70fc\") " pod="openshift-image-registry/node-ca-wwhph"
Sep 15 21:26:04.605326 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605317    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-run-ovn-kubernetes\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605333    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-cni-netd\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605361    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4eb4bf49-c243-426b-a143-22c987695db4-ovnkube-script-lib\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605377    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret\") pod \"global-pull-secret-syncer-v6j25\" (UID: \"21784d6c-c661-42eb-8b09-41376dacc1a3\") " pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:04.605686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605391    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/722562f8-7296-47bd-8fec-1d6791c5afd0-agent-certs\") pod \"konnectivity-agent-zvtxn\" (UID: \"722562f8-7296-47bd-8fec-1d6791c5afd0\") " pod="kube-system/konnectivity-agent-zvtxn"
Sep 15 21:26:04.605686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605407    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-systemd-units\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605421    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-node-log\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605439    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b3f47d0-ec5b-4699-91a9-160ea99f70fc-host\") pod \"node-ca-wwhph\" (UID: \"6b3f47d0-ec5b-4699-91a9-160ea99f70fc\") " pod="openshift-image-registry/node-ca-wwhph"
Sep 15 21:26:04.605686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605463    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-run-netns\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605481    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.605686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605503    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-log-socket\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605539    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4eb4bf49-c243-426b-a143-22c987695db4-env-overrides\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605565    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4eb4bf49-c243-426b-a143-22c987695db4-ovn-node-metrics-cert\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605592    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqn4c\" (UniqueName: \"kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c\") pod \"network-check-target-rsbdn\" (UID: \"f508cca8-4d27-4ee5-926d-1a7a3ce42286\") " pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:04.605686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605641    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-var-lib-openvswitch\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.605686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.605665    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4eb4bf49-c243-426b-a143-22c987695db4-ovnkube-config\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.607953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.606713    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 21:26:04.607953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.606734    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 21:26:04.607953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.606914    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 21:26:04.607953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.606988    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 21:26:04.607953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.606994    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 21:26:04.607953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.607525    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 21:26:04.607953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.607594    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 21:26:04.607953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.607905    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nzw9t\""
Sep 15 21:26:04.608327 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.607957    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 21:26:04.608327 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.607907    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7nrtm\""
Sep 15 21:26:04.608327 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.608319    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.610320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.610301    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:26:04.610825 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.610611    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mb6wk\""
Sep 15 21:26:04.610825 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.610640    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 21:26:04.610957 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.610921    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.613322 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.613301    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wfls6\""
Sep 15 21:26:04.613419 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.613390    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:04.613485 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:04.613467    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:04.613604 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.613584    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 21:26:04.615736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.615720    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tjsqw"
Sep 15 21:26:04.617983 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.617964    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:26:04.618168 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.618137    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pb8bj\""
Sep 15 21:26:04.618266 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.618169    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 21:26:04.618266 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.618209    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 21:26:04.639479 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.639455    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 21:26:04.639661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.639643    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zk5fw"
Sep 15 21:26:04.645264 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.645243    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zk5fw"
Sep 15 21:26:04.696483 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.696456    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 21:26:04.706290 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706264    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-sysctl-d\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.706415 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706299    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-var-lib-cni-multus\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.706415 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706317    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83ec8a70-a84e-4da5-a0db-f51c03a2dac1-host-slash\") pod \"iptables-alerter-tjsqw\" (UID: \"83ec8a70-a84e-4da5-a0db-f51c03a2dac1\") " pod="openshift-network-operator/iptables-alerter-tjsqw"
Sep 15 21:26:04.706415 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706338    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/722562f8-7296-47bd-8fec-1d6791c5afd0-konnectivity-ca\") pod \"konnectivity-agent-zvtxn\" (UID: \"722562f8-7296-47bd-8fec-1d6791c5afd0\") " pod="kube-system/konnectivity-agent-zvtxn"
Sep 15 21:26:04.706415 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706369    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-run-systemd\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.706415 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706395    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pzvz\" (UniqueName: \"kubernetes.io/projected/4eb4bf49-c243-426b-a143-22c987695db4-kube-api-access-5pzvz\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.706415 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706415    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-sys-fs\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.706723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706508    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-run-systemd\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.706723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706561    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-etc-openvswitch\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.706723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706603    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-run-ovn\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.706723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706631    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwxtx\" (UniqueName: \"kubernetes.io/projected/895233b3-517c-4b44-9dc1-02cb2e4ff8ac-kube-api-access-bwxtx\") pod \"node-resolver-rwmx9\" (UID: \"895233b3-517c-4b44-9dc1-02cb2e4ff8ac\") " pod="openshift-dns/node-resolver-rwmx9"
Sep 15 21:26:04.706723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706649    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs\") pod \"network-metrics-daemon-b46fs\" (UID: \"117891c4-3453-4091-81f0-46b5532f49b8\") " pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:04.706723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706668    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rmk6\" (UniqueName: \"kubernetes.io/projected/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-kube-api-access-9rmk6\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.706723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706645    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-etc-openvswitch\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.706723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706696    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-run-ovn\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.706723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706696    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxwnx\" (UniqueName: \"kubernetes.io/projected/6b3f47d0-ec5b-4699-91a9-160ea99f70fc-kube-api-access-bxwnx\") pod \"node-ca-wwhph\" (UID: \"6b3f47d0-ec5b-4699-91a9-160ea99f70fc\") " pod="openshift-image-registry/node-ca-wwhph"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706828    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-kubelet\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706848    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-cni-bin\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706865    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-cnibin\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706883    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-var-lib-cni-bin\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706883    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-kubelet\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706900    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706935    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-socket-dir\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706944    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-cni-bin\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706974    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6b3f47d0-ec5b-4699-91a9-160ea99f70fc-serviceca\") pod \"node-ca-wwhph\" (UID: \"6b3f47d0-ec5b-4699-91a9-160ea99f70fc\") " pod="openshift-image-registry/node-ca-wwhph"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.706979    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/722562f8-7296-47bd-8fec-1d6791c5afd0-konnectivity-ca\") pod \"konnectivity-agent-zvtxn\" (UID: \"722562f8-7296-47bd-8fec-1d6791c5afd0\") " pod="kube-system/konnectivity-agent-zvtxn"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707000    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-run-ovn-kubernetes\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707029    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-cni-netd\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4eb4bf49-c243-426b-a143-22c987695db4-ovnkube-script-lib\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707061    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-run-ovn-kubernetes\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707101    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-cni-netd\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707138    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret\") pod \"global-pull-secret-syncer-v6j25\" (UID: \"21784d6c-c661-42eb-8b09-41376dacc1a3\") " pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:04.707177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707169    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/895233b3-517c-4b44-9dc1-02cb2e4ff8ac-hosts-file\") pod \"node-resolver-rwmx9\" (UID: \"895233b3-517c-4b44-9dc1-02cb2e4ff8ac\") " pod="openshift-dns/node-resolver-rwmx9"
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707192    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2046dd47-2f92-4a60-a947-6d6ea2b62b83-cni-binary-copy\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707215    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-host\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707240    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-multus-conf-dir\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707274    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-cnibin\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707309    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-run-netns\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:04.707310    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707353    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-log-socket\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707379    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-run-netns\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:04.707408    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret podName:21784d6c-c661-42eb-8b09-41376dacc1a3 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:05.207364849 +0000 UTC m=+3.019373304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret") pod "global-pull-secret-syncer-v6j25" (UID: "21784d6c-c661-42eb-8b09-41376dacc1a3") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707416    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6b3f47d0-ec5b-4699-91a9-160ea99f70fc-serviceca\") pod \"node-ca-wwhph\" (UID: \"6b3f47d0-ec5b-4699-91a9-160ea99f70fc\") " pod="openshift-image-registry/node-ca-wwhph"
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707429    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-log-socket\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707482    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4eb4bf49-c243-426b-a143-22c987695db4-ovn-node-metrics-cert\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707514    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqtxq\" (UniqueName: \"kubernetes.io/projected/117891c4-3453-4091-81f0-46b5532f49b8-kube-api-access-qqtxq\") pod \"network-metrics-daemon-b46fs\" (UID: \"117891c4-3453-4091-81f0-46b5532f49b8\") " pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707538    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707555    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-modprobe-d\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707569    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-systemd\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.708088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707585    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83ec8a70-a84e-4da5-a0db-f51c03a2dac1-iptables-alerter-script\") pod \"iptables-alerter-tjsqw\" (UID: \"83ec8a70-a84e-4da5-a0db-f51c03a2dac1\") " pod="openshift-network-operator/iptables-alerter-tjsqw"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707613    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-run-openvswitch\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707633    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21784d6c-c661-42eb-8b09-41376dacc1a3-dbus\") pod \"global-pull-secret-syncer-v6j25\" (UID: \"21784d6c-c661-42eb-8b09-41376dacc1a3\") " pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707629    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4eb4bf49-c243-426b-a143-22c987695db4-ovnkube-script-lib\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707654    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-run-openvswitch\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707688    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895233b3-517c-4b44-9dc1-02cb2e4ff8ac-tmp-dir\") pod \"node-resolver-rwmx9\" (UID: \"895233b3-517c-4b44-9dc1-02cb2e4ff8ac\") " pod="openshift-dns/node-resolver-rwmx9"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707727    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/21784d6c-c661-42eb-8b09-41376dacc1a3-dbus\") pod \"global-pull-secret-syncer-v6j25\" (UID: \"21784d6c-c661-42eb-8b09-41376dacc1a3\") " pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707732    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-system-cni-dir\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707792    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-run-netns\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707811    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707832    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-sysconfig\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707870    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-slash\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707903    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707930    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-multus-socket-dir-parent\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707956    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-run-k8s-cni-cncf-io\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707972    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-slash\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707977    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-hostroot\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.708953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.707991    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-run-multus-certs\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708055    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-sysctl-conf\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-lib-modules\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708146    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21784d6c-c661-42eb-8b09-41376dacc1a3-kubelet-config\") pod \"global-pull-secret-syncer-v6j25\" (UID: \"21784d6c-c661-42eb-8b09-41376dacc1a3\") " pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708172    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-var-lib-kubelet\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnpp9\" (UniqueName: \"kubernetes.io/projected/2046dd47-2f92-4a60-a947-6d6ea2b62b83-kube-api-access-qnpp9\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708243    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/21784d6c-c661-42eb-8b09-41376dacc1a3-kubelet-config\") pod \"global-pull-secret-syncer-v6j25\" (UID: \"21784d6c-c661-42eb-8b09-41376dacc1a3\") " pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708291    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-registration-dir\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708312    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708319    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-multus-cni-dir\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708372    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2046dd47-2f92-4a60-a947-6d6ea2b62b83-multus-daemon-config\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708399    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-etc-kubernetes\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708423    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-run\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708446    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-sys\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708489    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-var-lib-kubelet\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.709590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708520    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c62v9\" (UniqueName: \"kubernetes.io/projected/58a3d8f9-f359-4579-960f-496389a95236-kube-api-access-c62v9\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/722562f8-7296-47bd-8fec-1d6791c5afd0-agent-certs\") pod \"konnectivity-agent-zvtxn\" (UID: \"722562f8-7296-47bd-8fec-1d6791c5afd0\") " pod="kube-system/konnectivity-agent-zvtxn"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708570    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-systemd-units\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708611    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-node-log\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708640    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh8fs\" (UniqueName: \"kubernetes.io/projected/83ec8a70-a84e-4da5-a0db-f51c03a2dac1-kube-api-access-lh8fs\") pod \"iptables-alerter-tjsqw\" (UID: \"83ec8a70-a84e-4da5-a0db-f51c03a2dac1\") " pod="openshift-network-operator/iptables-alerter-tjsqw"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708665    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58a3d8f9-f359-4579-960f-496389a95236-tmp\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708668    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-systemd-units\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b3f47d0-ec5b-4699-91a9-160ea99f70fc-host\") pod \"node-ca-wwhph\" (UID: \"6b3f47d0-ec5b-4699-91a9-160ea99f70fc\") " pod="openshift-image-registry/node-ca-wwhph"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4eb4bf49-c243-426b-a143-22c987695db4-env-overrides\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708749    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-system-cni-dir\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708765    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708780    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-node-log\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708804    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-device-dir\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708812    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b3f47d0-ec5b-4699-91a9-160ea99f70fc-host\") pod \"node-ca-wwhph\" (UID: \"6b3f47d0-ec5b-4699-91a9-160ea99f70fc\") " pod="openshift-image-registry/node-ca-wwhph"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708829    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708859    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58a3d8f9-f359-4579-960f-496389a95236-etc-tuned\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708886    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqn4c\" (UniqueName: \"kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c\") pod \"network-check-target-rsbdn\" (UID: \"f508cca8-4d27-4ee5-926d-1a7a3ce42286\") " pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:04.710305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708919    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-var-lib-openvswitch\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.711089 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.708965    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4eb4bf49-c243-426b-a143-22c987695db4-ovnkube-config\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.711089 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.709001    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx2c6\" (UniqueName: \"kubernetes.io/projected/488a7507-e77a-4520-8855-4ec61e80c149-kube-api-access-dx2c6\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.711089 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.709054    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-kubernetes\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.711089 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.709114    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4eb4bf49-c243-426b-a143-22c987695db4-env-overrides\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.711089 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.709158    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4eb4bf49-c243-426b-a143-22c987695db4-var-lib-openvswitch\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.711089 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.709575    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4eb4bf49-c243-426b-a143-22c987695db4-ovnkube-config\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.712969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.712948    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4eb4bf49-c243-426b-a143-22c987695db4-ovn-node-metrics-cert\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.713248 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.713229    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/722562f8-7296-47bd-8fec-1d6791c5afd0-agent-certs\") pod \"konnectivity-agent-zvtxn\" (UID: \"722562f8-7296-47bd-8fec-1d6791c5afd0\") " pod="kube-system/konnectivity-agent-zvtxn"
Sep 15 21:26:04.716026 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:04.715999    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:26:04.716141 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:04.716035    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:26:04.716141 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:04.716049    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qqn4c for pod openshift-network-diagnostics/network-check-target-rsbdn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:26:04.716141 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:04.716121    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c podName:f508cca8-4d27-4ee5-926d-1a7a3ce42286 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:05.216102491 +0000 UTC m=+3.028110929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qqn4c" (UniqueName: "kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c") pod "network-check-target-rsbdn" (UID: "f508cca8-4d27-4ee5-926d-1a7a3ce42286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:26:04.718187 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.718164    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pzvz\" (UniqueName: \"kubernetes.io/projected/4eb4bf49-c243-426b-a143-22c987695db4-kube-api-access-5pzvz\") pod \"ovnkube-node-mpwmh\" (UID: \"4eb4bf49-c243-426b-a143-22c987695db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.720060 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.720033    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxwnx\" (UniqueName: \"kubernetes.io/projected/6b3f47d0-ec5b-4699-91a9-160ea99f70fc-kube-api-access-bxwnx\") pod \"node-ca-wwhph\" (UID: \"6b3f47d0-ec5b-4699-91a9-160ea99f70fc\") " pod="openshift-image-registry/node-ca-wwhph"
Sep 15 21:26:04.746030 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.745976    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" event={"ID":"ea4e7f0d7b04a6dd59e41a5028a23abd","Type":"ContainerStarted","Data":"74f9f9a042ddead316735970633a125eea44550bdbed44466c59486e362cdd69"}
Sep 15 21:26:04.747296 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.747273    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" event={"ID":"2362bd6974628566f33836616227d001","Type":"ContainerStarted","Data":"dbea59e7dcd04babb86b450fca43e1d2d7b9de56c00e052486b9f8358bf2a5e0"}
Sep 15 21:26:04.809484 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809386    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-sys\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.809484 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809426    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-var-lib-kubelet\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.809484 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809451    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c62v9\" (UniqueName: \"kubernetes.io/projected/58a3d8f9-f359-4579-960f-496389a95236-kube-api-access-c62v9\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.809484 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8fs\" (UniqueName: \"kubernetes.io/projected/83ec8a70-a84e-4da5-a0db-f51c03a2dac1-kube-api-access-lh8fs\") pod \"iptables-alerter-tjsqw\" (UID: \"83ec8a70-a84e-4da5-a0db-f51c03a2dac1\") " pod="openshift-network-operator/iptables-alerter-tjsqw"
Sep 15 21:26:04.809484 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809482    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-sys\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809493    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-var-lib-kubelet\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809489    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58a3d8f9-f359-4579-960f-496389a95236-tmp\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809552    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-system-cni-dir\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809576    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809596    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-device-dir\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809620    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809646    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58a3d8f9-f359-4579-960f-496389a95236-etc-tuned\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809666    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-device-dir\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809689    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx2c6\" (UniqueName: \"kubernetes.io/projected/488a7507-e77a-4520-8855-4ec61e80c149-kube-api-access-dx2c6\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809715    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-kubernetes\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809752    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-sysctl-d\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-var-lib-cni-multus\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809821    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83ec8a70-a84e-4da5-a0db-f51c03a2dac1-host-slash\") pod \"iptables-alerter-tjsqw\" (UID: \"83ec8a70-a84e-4da5-a0db-f51c03a2dac1\") " pod="openshift-network-operator/iptables-alerter-tjsqw"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809832    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809849    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-sys-fs\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809890    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwxtx\" (UniqueName: \"kubernetes.io/projected/895233b3-517c-4b44-9dc1-02cb2e4ff8ac-kube-api-access-bwxtx\") pod \"node-resolver-rwmx9\" (UID: \"895233b3-517c-4b44-9dc1-02cb2e4ff8ac\") " pod="openshift-dns/node-resolver-rwmx9"
Sep 15 21:26:04.809924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809897    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-kubernetes\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809918    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs\") pod \"network-metrics-daemon-b46fs\" (UID: \"117891c4-3453-4091-81f0-46b5532f49b8\") " pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809900    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-sys-fs\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809944    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rmk6\" (UniqueName: \"kubernetes.io/projected/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-kube-api-access-9rmk6\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.809979    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-cnibin\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810005    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-var-lib-cni-bin\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810034    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810045    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-var-lib-cni-multus\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810062    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-socket-dir\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810104    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/895233b3-517c-4b44-9dc1-02cb2e4ff8ac-hosts-file\") pod \"node-resolver-rwmx9\" (UID: \"895233b3-517c-4b44-9dc1-02cb2e4ff8ac\") " pod="openshift-dns/node-resolver-rwmx9"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810091    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83ec8a70-a84e-4da5-a0db-f51c03a2dac1-host-slash\") pod \"iptables-alerter-tjsqw\" (UID: \"83ec8a70-a84e-4da5-a0db-f51c03a2dac1\") " pod="openshift-network-operator/iptables-alerter-tjsqw"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810150    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2046dd47-2f92-4a60-a947-6d6ea2b62b83-cni-binary-copy\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810162    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-cnibin\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810175    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-host\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810180    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/895233b3-517c-4b44-9dc1-02cb2e4ff8ac-hosts-file\") pod \"node-resolver-rwmx9\" (UID: \"895233b3-517c-4b44-9dc1-02cb2e4ff8ac\") " pod="openshift-dns/node-resolver-rwmx9"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810206    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-multus-conf-dir\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810245    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-cnibin\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.810853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810279    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqtxq\" (UniqueName: \"kubernetes.io/projected/117891c4-3453-4091-81f0-46b5532f49b8-kube-api-access-qqtxq\") pod \"network-metrics-daemon-b46fs\" (UID: \"117891c4-3453-4091-81f0-46b5532f49b8\") " pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810281    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810314    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:04.810326    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810369    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-system-cni-dir\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810383    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-host\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810004    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-sysctl-d\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810412    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-multus-conf-dir\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810455    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-cnibin\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810456    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-var-lib-cni-bin\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810479    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-modprobe-d\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:04.810504    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs podName:117891c4-3453-4091-81f0-46b5532f49b8 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:05.310485758 +0000 UTC m=+3.122494182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs") pod "network-metrics-daemon-b46fs" (UID: "117891c4-3453-4091-81f0-46b5532f49b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810537    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-systemd\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810559    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-modprobe-d\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810566    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83ec8a70-a84e-4da5-a0db-f51c03a2dac1-iptables-alerter-script\") pod \"iptables-alerter-tjsqw\" (UID: \"83ec8a70-a84e-4da5-a0db-f51c03a2dac1\") " pod="openshift-network-operator/iptables-alerter-tjsqw"
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810583    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-socket-dir\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810598    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895233b3-517c-4b44-9dc1-02cb2e4ff8ac-tmp-dir\") pod \"node-resolver-rwmx9\" (UID: \"895233b3-517c-4b44-9dc1-02cb2e4ff8ac\") " pod="openshift-dns/node-resolver-rwmx9"
Sep 15 21:26:04.811698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810610    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810624    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-system-cni-dir\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810650    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-run-netns\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810676    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810703    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-sysconfig\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810738    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-multus-socket-dir-parent\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810765    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-run-k8s-cni-cncf-io\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810774    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-system-cni-dir\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810791    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-hostroot\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810820    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-run-multus-certs\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810861    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-sysctl-conf\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810888    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-lib-modules\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-var-lib-kubelet\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810942    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnpp9\" (UniqueName: \"kubernetes.io/projected/2046dd47-2f92-4a60-a947-6d6ea2b62b83-kube-api-access-qnpp9\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.810972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811002    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-registration-dir\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-multus-cni-dir\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.812487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811032    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-cni-binary-copy\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811054    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2046dd47-2f92-4a60-a947-6d6ea2b62b83-multus-daemon-config\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811078    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-etc-kubernetes\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811101    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-systemd\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811103    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-run\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811129    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/895233b3-517c-4b44-9dc1-02cb2e4ff8ac-tmp-dir\") pod \"node-resolver-rwmx9\" (UID: \"895233b3-517c-4b44-9dc1-02cb2e4ff8ac\") " pod="openshift-dns/node-resolver-rwmx9"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811155    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-run\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811240    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-registration-dir\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811251    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-lib-modules\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811291    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-var-lib-kubelet\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811317    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-multus-cni-dir\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83ec8a70-a84e-4da5-a0db-f51c03a2dac1-iptables-alerter-script\") pod \"iptables-alerter-tjsqw\" (UID: \"83ec8a70-a84e-4da5-a0db-f51c03a2dac1\") " pod="openshift-network-operator/iptables-alerter-tjsqw"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811365    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/488a7507-e77a-4520-8855-4ec61e80c149-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811386    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2046dd47-2f92-4a60-a947-6d6ea2b62b83-cni-binary-copy\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811291    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-sysctl-conf\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811421    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/58a3d8f9-f359-4579-960f-496389a95236-etc-sysconfig\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811474    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-run-k8s-cni-cncf-io\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811476    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-etc-kubernetes\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.813345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811538    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-run-multus-certs\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.813906 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811539    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-hostroot\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.813906 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811583    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-multus-socket-dir-parent\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.813906 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811631    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2046dd47-2f92-4a60-a947-6d6ea2b62b83-host-run-netns\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.813906 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811824    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.813906 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.811823    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2046dd47-2f92-4a60-a947-6d6ea2b62b83-multus-daemon-config\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.813906 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.812629    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58a3d8f9-f359-4579-960f-496389a95236-tmp\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.813906 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.813029    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/58a3d8f9-f359-4579-960f-496389a95236-etc-tuned\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.822607 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.822582    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c62v9\" (UniqueName: \"kubernetes.io/projected/58a3d8f9-f359-4579-960f-496389a95236-kube-api-access-c62v9\") pod \"tuned-lcz99\" (UID: \"58a3d8f9-f359-4579-960f-496389a95236\") " pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.822723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.822611    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh8fs\" (UniqueName: \"kubernetes.io/projected/83ec8a70-a84e-4da5-a0db-f51c03a2dac1-kube-api-access-lh8fs\") pod \"iptables-alerter-tjsqw\" (UID: \"83ec8a70-a84e-4da5-a0db-f51c03a2dac1\") " pod="openshift-network-operator/iptables-alerter-tjsqw"
Sep 15 21:26:04.822811 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.822774    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwxtx\" (UniqueName: \"kubernetes.io/projected/895233b3-517c-4b44-9dc1-02cb2e4ff8ac-kube-api-access-bwxtx\") pod \"node-resolver-rwmx9\" (UID: \"895233b3-517c-4b44-9dc1-02cb2e4ff8ac\") " pod="openshift-dns/node-resolver-rwmx9"
Sep 15 21:26:04.822939 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.822922    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx2c6\" (UniqueName: \"kubernetes.io/projected/488a7507-e77a-4520-8855-4ec61e80c149-kube-api-access-dx2c6\") pod \"aws-ebs-csi-driver-node-v8trf\" (UID: \"488a7507-e77a-4520-8855-4ec61e80c149\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.827063 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.827043    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rmk6\" (UniqueName: \"kubernetes.io/projected/2d221fc3-4447-44fe-bf1e-6bb0890ce88a-kube-api-access-9rmk6\") pod \"multus-additional-cni-plugins-jmmgp\" (UID: \"2d221fc3-4447-44fe-bf1e-6bb0890ce88a\") " pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.827428 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.827408    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqtxq\" (UniqueName: \"kubernetes.io/projected/117891c4-3453-4091-81f0-46b5532f49b8-kube-api-access-qqtxq\") pod \"network-metrics-daemon-b46fs\" (UID: \"117891c4-3453-4091-81f0-46b5532f49b8\") " pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:04.830681 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.830665    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnpp9\" (UniqueName: \"kubernetes.io/projected/2046dd47-2f92-4a60-a947-6d6ea2b62b83-kube-api-access-qnpp9\") pod \"multus-js9vq\" (UID: \"2046dd47-2f92-4a60-a947-6d6ea2b62b83\") " pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.902925 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.902886    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wwhph"
Sep 15 21:26:04.912196 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.911951    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zvtxn"
Sep 15 21:26:04.918688 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:04.918659    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3f47d0_ec5b_4699_91a9_160ea99f70fc.slice/crio-6fc2e333d68b0988eb6765e1abb777a3a6661d157fb2a6ebcf5caf4793107d75 WatchSource:0}: Error finding container 6fc2e333d68b0988eb6765e1abb777a3a6661d157fb2a6ebcf5caf4793107d75: Status 404 returned error can't find the container with id 6fc2e333d68b0988eb6765e1abb777a3a6661d157fb2a6ebcf5caf4793107d75
Sep 15 21:26:04.924151 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:04.924123    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722562f8_7296_47bd_8fec_1d6791c5afd0.slice/crio-11fcca52bcac511a46d56c4e965ea466d1c1177a3609ead61cc7a50a99d64094 WatchSource:0}: Error finding container 11fcca52bcac511a46d56c4e965ea466d1c1177a3609ead61cc7a50a99d64094: Status 404 returned error can't find the container with id 11fcca52bcac511a46d56c4e965ea466d1c1177a3609ead61cc7a50a99d64094
Sep 15 21:26:04.924933 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.924914    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:04.931835 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.931811    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rwmx9"
Sep 15 21:26:04.935292 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:04.935272    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb4bf49_c243_426b_a143_22c987695db4.slice/crio-d842df021a8409d8e8155f88b487df4141d9121dd56afb143865bc5f205a1114 WatchSource:0}: Error finding container d842df021a8409d8e8155f88b487df4141d9121dd56afb143865bc5f205a1114: Status 404 returned error can't find the container with id d842df021a8409d8e8155f88b487df4141d9121dd56afb143865bc5f205a1114
Sep 15 21:26:04.939989 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.939971    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jmmgp"
Sep 15 21:26:04.940877 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:04.940853    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895233b3_517c_4b44_9dc1_02cb2e4ff8ac.slice/crio-554759483a2e098984bd3d4c05bf6be0e57a964e174319a37f019e470553fda2 WatchSource:0}: Error finding container 554759483a2e098984bd3d4c05bf6be0e57a964e174319a37f019e470553fda2: Status 404 returned error can't find the container with id 554759483a2e098984bd3d4c05bf6be0e57a964e174319a37f019e470553fda2
Sep 15 21:26:04.948042 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.948025    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf"
Sep 15 21:26:04.949277 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:04.949258    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d221fc3_4447_44fe_bf1e_6bb0890ce88a.slice/crio-cac17dfbf30b33c8e4b28b8f77ae5a2dbc784997e01f0648f1fdceb74c463f57 WatchSource:0}: Error finding container cac17dfbf30b33c8e4b28b8f77ae5a2dbc784997e01f0648f1fdceb74c463f57: Status 404 returned error can't find the container with id cac17dfbf30b33c8e4b28b8f77ae5a2dbc784997e01f0648f1fdceb74c463f57
Sep 15 21:26:04.955848 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.955827    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lcz99"
Sep 15 21:26:04.956864 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:04.956825    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod488a7507_e77a_4520_8855_4ec61e80c149.slice/crio-570b1fe87a6d36e2a629c2d4d56894b5acbe979259c6240bcdf37be4d92afc5f WatchSource:0}: Error finding container 570b1fe87a6d36e2a629c2d4d56894b5acbe979259c6240bcdf37be4d92afc5f: Status 404 returned error can't find the container with id 570b1fe87a6d36e2a629c2d4d56894b5acbe979259c6240bcdf37be4d92afc5f
Sep 15 21:26:04.964065 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.964046    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-js9vq"
Sep 15 21:26:04.964130 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:04.964076    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a3d8f9_f359_4579_960f_496389a95236.slice/crio-95d669574dac6fd99006f89c9f8ec49642df6c088bbb2c2868eb0188b0c9e930 WatchSource:0}: Error finding container 95d669574dac6fd99006f89c9f8ec49642df6c088bbb2c2868eb0188b0c9e930: Status 404 returned error can't find the container with id 95d669574dac6fd99006f89c9f8ec49642df6c088bbb2c2868eb0188b0c9e930
Sep 15 21:26:04.971847 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:04.971824    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tjsqw"
Sep 15 21:26:04.972194 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:04.972176    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2046dd47_2f92_4a60_a947_6d6ea2b62b83.slice/crio-6ecb121aa9fc778bb1a734d9825aabfa4dfff0c915ff4fd2fd1c2012d836c910 WatchSource:0}: Error finding container 6ecb121aa9fc778bb1a734d9825aabfa4dfff0c915ff4fd2fd1c2012d836c910: Status 404 returned error can't find the container with id 6ecb121aa9fc778bb1a734d9825aabfa4dfff0c915ff4fd2fd1c2012d836c910
Sep 15 21:26:04.980738 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:04.980720    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ec8a70_a84e_4da5_a0db_f51c03a2dac1.slice/crio-81ae8e67274e3f3ca811ae85c0968baa12602ca21dfb67c9437905c32f61e838 WatchSource:0}: Error finding container 81ae8e67274e3f3ca811ae85c0968baa12602ca21dfb67c9437905c32f61e838: Status 404 returned error can't find the container with id 81ae8e67274e3f3ca811ae85c0968baa12602ca21dfb67c9437905c32f61e838
Sep 15 21:26:05.215054 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.215017    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret\") pod \"global-pull-secret-syncer-v6j25\" (UID: \"21784d6c-c661-42eb-8b09-41376dacc1a3\") " pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:05.215264 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:05.215190    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:26:05.215326 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:05.215270    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret podName:21784d6c-c661-42eb-8b09-41376dacc1a3 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:06.215251551 +0000 UTC m=+4.027259975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret") pod "global-pull-secret-syncer-v6j25" (UID: "21784d6c-c661-42eb-8b09-41376dacc1a3") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:26:05.316520 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.316481    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqn4c\" (UniqueName: \"kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c\") pod \"network-check-target-rsbdn\" (UID: \"f508cca8-4d27-4ee5-926d-1a7a3ce42286\") " pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:05.316725 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.316542    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs\") pod \"network-metrics-daemon-b46fs\" (UID: \"117891c4-3453-4091-81f0-46b5532f49b8\") " pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:05.316725 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:05.316671    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:26:05.316725 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:05.316677    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:26:05.316725 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:05.316700    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:26:05.316725 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:05.316715    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qqn4c for pod openshift-network-diagnostics/network-check-target-rsbdn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:26:05.317040 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:05.316761    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs podName:117891c4-3453-4091-81f0-46b5532f49b8 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:06.316744726 +0000 UTC m=+4.128753150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs") pod "network-metrics-daemon-b46fs" (UID: "117891c4-3453-4091-81f0-46b5532f49b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:26:05.317040 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:05.316778    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c podName:f508cca8-4d27-4ee5-926d-1a7a3ce42286 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:06.316771675 +0000 UTC m=+4.128780100 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqn4c" (UniqueName: "kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c") pod "network-check-target-rsbdn" (UID: "f508cca8-4d27-4ee5-926d-1a7a3ce42286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:26:05.641004 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.640756    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 21:26:05.650875 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.650716    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:21:05 +0000 UTC" deadline="2028-02-12 15:40:27.437697175 +0000 UTC"
Sep 15 21:26:05.650875 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.650785    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12354h14m21.786916242s"
Sep 15 21:26:05.741084 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.741052    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:05.741265 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:05.741207    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:05.741660 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.741640    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:05.741780 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:05.741757    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:05.754256 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.754224    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tjsqw" event={"ID":"83ec8a70-a84e-4da5-a0db-f51c03a2dac1","Type":"ContainerStarted","Data":"81ae8e67274e3f3ca811ae85c0968baa12602ca21dfb67c9437905c32f61e838"}
Sep 15 21:26:05.758887 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.758862    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmmgp" event={"ID":"2d221fc3-4447-44fe-bf1e-6bb0890ce88a","Type":"ContainerStarted","Data":"cac17dfbf30b33c8e4b28b8f77ae5a2dbc784997e01f0648f1fdceb74c463f57"}
Sep 15 21:26:05.768405 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.768356    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rwmx9" event={"ID":"895233b3-517c-4b44-9dc1-02cb2e4ff8ac","Type":"ContainerStarted","Data":"554759483a2e098984bd3d4c05bf6be0e57a964e174319a37f019e470553fda2"}
Sep 15 21:26:05.773207 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.773149    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zvtxn" event={"ID":"722562f8-7296-47bd-8fec-1d6791c5afd0","Type":"ContainerStarted","Data":"11fcca52bcac511a46d56c4e965ea466d1c1177a3609ead61cc7a50a99d64094"}
Sep 15 21:26:05.775162 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.775108    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-js9vq" event={"ID":"2046dd47-2f92-4a60-a947-6d6ea2b62b83","Type":"ContainerStarted","Data":"6ecb121aa9fc778bb1a734d9825aabfa4dfff0c915ff4fd2fd1c2012d836c910"}
Sep 15 21:26:05.778460 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.778388    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lcz99" event={"ID":"58a3d8f9-f359-4579-960f-496389a95236","Type":"ContainerStarted","Data":"95d669574dac6fd99006f89c9f8ec49642df6c088bbb2c2868eb0188b0c9e930"}
Sep 15 21:26:05.781460 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.781390    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf" event={"ID":"488a7507-e77a-4520-8855-4ec61e80c149","Type":"ContainerStarted","Data":"570b1fe87a6d36e2a629c2d4d56894b5acbe979259c6240bcdf37be4d92afc5f"}
Sep 15 21:26:05.783590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.783563    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh" event={"ID":"4eb4bf49-c243-426b-a143-22c987695db4","Type":"ContainerStarted","Data":"d842df021a8409d8e8155f88b487df4141d9121dd56afb143865bc5f205a1114"}
Sep 15 21:26:05.785678 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:05.785653    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wwhph" event={"ID":"6b3f47d0-ec5b-4699-91a9-160ea99f70fc","Type":"ContainerStarted","Data":"6fc2e333d68b0988eb6765e1abb777a3a6661d157fb2a6ebcf5caf4793107d75"}
Sep 15 21:26:06.223260 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:06.223226    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret\") pod \"global-pull-secret-syncer-v6j25\" (UID: \"21784d6c-c661-42eb-8b09-41376dacc1a3\") " pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:06.223469 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:06.223445    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:26:06.223556 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:06.223532    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret podName:21784d6c-c661-42eb-8b09-41376dacc1a3 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:08.223506607 +0000 UTC m=+6.035515045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret") pod "global-pull-secret-syncer-v6j25" (UID: "21784d6c-c661-42eb-8b09-41376dacc1a3") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:26:06.324635 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:06.324544    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs\") pod \"network-metrics-daemon-b46fs\" (UID: \"117891c4-3453-4091-81f0-46b5532f49b8\") " pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:06.324816 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:06.324651    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqn4c\" (UniqueName: \"kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c\") pod \"network-check-target-rsbdn\" (UID: \"f508cca8-4d27-4ee5-926d-1a7a3ce42286\") " pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:06.324816 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:06.324720    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:26:06.324816 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:06.324792    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:26:06.325040 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:06.324832    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:26:06.325040 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:06.324848    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qqn4c for pod openshift-network-diagnostics/network-check-target-rsbdn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:26:06.325040 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:06.324815    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs podName:117891c4-3453-4091-81f0-46b5532f49b8 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:08.324793099 +0000 UTC m=+6.136801540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs") pod "network-metrics-daemon-b46fs" (UID: "117891c4-3453-4091-81f0-46b5532f49b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:26:06.325040 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:06.324915    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c podName:f508cca8-4d27-4ee5-926d-1a7a3ce42286 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:08.324898384 +0000 UTC m=+6.136906816 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqn4c" (UniqueName: "kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c") pod "network-check-target-rsbdn" (UID: "f508cca8-4d27-4ee5-926d-1a7a3ce42286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:26:06.651569 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:06.651517    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 21:21:05 +0000 UTC" deadline="2028-05-22 02:19:53.307947281 +0000 UTC"
Sep 15 21:26:06.651569 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:06.651567    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14740h53m46.65638505s"
Sep 15 21:26:06.748005 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:06.747508    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:06.748005 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:06.747636    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:06.796933 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:06.796896    2575 generic.go:356] "Generic (PLEG): container finished" podID="2362bd6974628566f33836616227d001" containerID="ec6c01a9cd5c138addbaef915baefbd805a2a5c236544e14e05c578d0dc51c12" exitCode=0
Sep 15 21:26:06.797101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:06.796954    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" event={"ID":"2362bd6974628566f33836616227d001","Type":"ContainerDied","Data":"ec6c01a9cd5c138addbaef915baefbd805a2a5c236544e14e05c578d0dc51c12"}
Sep 15 21:26:07.741561 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:07.741517    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:07.742241 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:07.741686    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:07.742241 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:07.742129    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:07.742376 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:07.742236    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:07.820690 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:07.820621    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" event={"ID":"2362bd6974628566f33836616227d001","Type":"ContainerStarted","Data":"38f4cf0b3a38da4440e391cba0d422d31596874fadfaf62f770c4c154a28c8b5"}
Sep 15 21:26:08.241887 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:08.241844    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret\") pod \"global-pull-secret-syncer-v6j25\" (UID: \"21784d6c-c661-42eb-8b09-41376dacc1a3\") " pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:08.242082 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:08.242053    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:26:08.242148 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:08.242119    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret podName:21784d6c-c661-42eb-8b09-41376dacc1a3 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:12.242099036 +0000 UTC m=+10.054107487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret") pod "global-pull-secret-syncer-v6j25" (UID: "21784d6c-c661-42eb-8b09-41376dacc1a3") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:26:08.344086 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:08.343205    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqn4c\" (UniqueName: \"kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c\") pod \"network-check-target-rsbdn\" (UID: \"f508cca8-4d27-4ee5-926d-1a7a3ce42286\") " pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:08.344086 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:08.343264    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs\") pod \"network-metrics-daemon-b46fs\" (UID: \"117891c4-3453-4091-81f0-46b5532f49b8\") " pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:08.344086 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:08.343430    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:26:08.344086 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:08.343498    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs podName:117891c4-3453-4091-81f0-46b5532f49b8 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:12.343477332 +0000 UTC m=+10.155485770 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs") pod "network-metrics-daemon-b46fs" (UID: "117891c4-3453-4091-81f0-46b5532f49b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:26:08.344086 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:08.343941    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:26:08.344086 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:08.343959    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:26:08.344086 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:08.343973    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qqn4c for pod openshift-network-diagnostics/network-check-target-rsbdn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:26:08.344086 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:08.344021    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c podName:f508cca8-4d27-4ee5-926d-1a7a3ce42286 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:12.344005321 +0000 UTC m=+10.156013759 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqn4c" (UniqueName: "kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c") pod "network-check-target-rsbdn" (UID: "f508cca8-4d27-4ee5-926d-1a7a3ce42286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:26:08.742993 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:08.742959    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:08.743384 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:08.743091    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:09.741231 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:09.741192    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:09.741443 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:09.741339    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:09.741761 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:09.741742    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:09.741912 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:09.741888    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:10.743041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:10.743002    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:10.743484 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:10.743159    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:11.741186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:11.741145    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:11.741392 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:11.741294    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:11.741459 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:11.741406    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:11.741520 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:11.741502    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:12.279813 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:12.279754    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret\") pod \"global-pull-secret-syncer-v6j25\" (UID: \"21784d6c-c661-42eb-8b09-41376dacc1a3\") " pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:12.280275 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:12.280008    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:26:12.280275 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:12.280096    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret podName:21784d6c-c661-42eb-8b09-41376dacc1a3 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:20.280073455 +0000 UTC m=+18.092081905 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret") pod "global-pull-secret-syncer-v6j25" (UID: "21784d6c-c661-42eb-8b09-41376dacc1a3") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:26:12.380841 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:12.380799    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqn4c\" (UniqueName: \"kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c\") pod \"network-check-target-rsbdn\" (UID: \"f508cca8-4d27-4ee5-926d-1a7a3ce42286\") " pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:12.381043 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:12.380857    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs\") pod \"network-metrics-daemon-b46fs\" (UID: \"117891c4-3453-4091-81f0-46b5532f49b8\") " pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:12.381043 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:12.380970    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:26:12.381043 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:12.381024    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs podName:117891c4-3453-4091-81f0-46b5532f49b8 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:20.381007073 +0000 UTC m=+18.193015511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs") pod "network-metrics-daemon-b46fs" (UID: "117891c4-3453-4091-81f0-46b5532f49b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:26:12.381477 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:12.381458    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:26:12.381556 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:12.381487    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:26:12.381556 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:12.381500    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qqn4c for pod openshift-network-diagnostics/network-check-target-rsbdn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:26:12.381556 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:12.381546    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c podName:f508cca8-4d27-4ee5-926d-1a7a3ce42286 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:20.381530179 +0000 UTC m=+18.193538621 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqn4c" (UniqueName: "kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c") pod "network-check-target-rsbdn" (UID: "f508cca8-4d27-4ee5-926d-1a7a3ce42286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:26:12.741934 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:12.741897    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:12.742112 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:12.741997    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:13.741419 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:13.741382    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:13.741926 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:13.741514    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:13.741926 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:13.741901    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:13.742039 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:13.742013    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:14.741137 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:14.741101    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:14.741329 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:14.741232    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:15.741579 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:15.741526    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:15.741579 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:15.741565    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:15.742086 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:15.741674    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:15.742086 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:15.741839    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:16.741734 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:16.741700    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:16.742220 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:16.741869    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:17.741873 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:17.741639    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:17.742354 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:17.741638    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:17.742354 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:17.742007    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:17.742354 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:17.742027    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:18.740744 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:18.740711    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:18.740961 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:18.740837    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:19.740919 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:19.740882    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:19.741361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:19.740929    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:19.741361 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:19.741008    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:19.741361 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:19.741131    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:20.340014 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:20.339981    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret\") pod \"global-pull-secret-syncer-v6j25\" (UID: \"21784d6c-c661-42eb-8b09-41376dacc1a3\") " pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:20.340284 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:20.340118    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:26:20.340284 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:20.340176    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret podName:21784d6c-c661-42eb-8b09-41376dacc1a3 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:36.340160955 +0000 UTC m=+34.152169397 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret") pod "global-pull-secret-syncer-v6j25" (UID: "21784d6c-c661-42eb-8b09-41376dacc1a3") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:26:20.440500 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:20.440454    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs\") pod \"network-metrics-daemon-b46fs\" (UID: \"117891c4-3453-4091-81f0-46b5532f49b8\") " pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:20.440664 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:20.440566    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqn4c\" (UniqueName: \"kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c\") pod \"network-check-target-rsbdn\" (UID: \"f508cca8-4d27-4ee5-926d-1a7a3ce42286\") " pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:20.440664 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:20.440609    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:26:20.440783 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:20.440693    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs podName:117891c4-3453-4091-81f0-46b5532f49b8 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:36.44066779 +0000 UTC m=+34.252676214 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs") pod "network-metrics-daemon-b46fs" (UID: "117891c4-3453-4091-81f0-46b5532f49b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:26:20.440783 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:20.440699    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:26:20.440783 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:20.440718    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:26:20.440783 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:20.440728    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qqn4c for pod openshift-network-diagnostics/network-check-target-rsbdn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:26:20.440957 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:20.440806    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c podName:f508cca8-4d27-4ee5-926d-1a7a3ce42286 nodeName:}" failed. No retries permitted until 2026-09-15 21:26:36.440793743 +0000 UTC m=+34.252802178 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqn4c" (UniqueName: "kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c") pod "network-check-target-rsbdn" (UID: "f508cca8-4d27-4ee5-926d-1a7a3ce42286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:26:20.741484 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:20.741441    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:20.741977 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:20.741594    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:21.741672 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:21.741630    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:21.742106 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:21.741677    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:21.742106 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:21.741786    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:21.742106 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:21.741842    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:22.741555 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:22.741519    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:22.741744 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:22.741637    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:23.740935 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:23.740900    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:23.741126 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:23.740900    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:23.741126 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:23.741020    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:23.741126 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:23.741114    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:24.748972 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:24.747679    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:24.748972 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:24.747851    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:24.861817 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:24.861731    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" event={"ID":"ea4e7f0d7b04a6dd59e41a5028a23abd","Type":"ContainerStarted","Data":"e15d15b2adcde8889e93180d8b4c5f46b1808a793f80427cd1fc136c1edf85e2"}
Sep 15 21:26:24.865967 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:24.865444    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lcz99" event={"ID":"58a3d8f9-f359-4579-960f-496389a95236","Type":"ContainerStarted","Data":"bc42c554790aa60b9f396c66714287fd99d84197329d88b6c9310a6d8539d478"}
Sep 15 21:26:24.875803 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:24.875707    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-148.ec2.internal" podStartSLOduration=20.875688515 podStartE2EDuration="20.875688515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:24.875338821 +0000 UTC m=+22.687347267" watchObservedRunningTime="2026-09-15 21:26:24.875688515 +0000 UTC m=+22.687696983"
Sep 15 21:26:24.876874 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:24.876395    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-148.ec2.internal" podStartSLOduration=20.876383194 podStartE2EDuration="20.876383194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:26:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:07.845700339 +0000 UTC m=+5.657708789" watchObservedRunningTime="2026-09-15 21:26:24.876383194 +0000 UTC m=+22.688391640"
Sep 15 21:26:24.891510 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:24.891471    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lcz99" podStartSLOduration=2.131691691 podStartE2EDuration="21.891455613s" totalImagesPullingTime="19.759763922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:24.890854235 +0000 UTC m=+22.702862679" watchObservedRunningTime="2026-09-15 21:26:24.891455613 +0000 UTC m=+22.703464063"
Sep 15 21:26:25.741060 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.740980    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:25.741240 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:25.741091    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:25.741240 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.741143    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:25.741240 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:25.741233    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:25.869750 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.869425    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-js9vq" event={"ID":"2046dd47-2f92-4a60-a947-6d6ea2b62b83","Type":"ContainerStarted","Data":"5ba589430fcef1f2146ac994607a9c1fc95a1aae2caad86d6d9709b9c40986c2"}
Sep 15 21:26:25.874415 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.874379    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf" event={"ID":"488a7507-e77a-4520-8855-4ec61e80c149","Type":"ContainerStarted","Data":"aaaef16fbe1b2ac3eb4f83ea55404bafac935f9766d5dc528d151dc8952514f9"}
Sep 15 21:26:25.877853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.877827    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh" event={"ID":"4eb4bf49-c243-426b-a143-22c987695db4","Type":"ContainerStarted","Data":"134a0be97a8ad33283f30fab5273b29c3c5dd736a3c30f98d82f0eb391973d45"}
Sep 15 21:26:25.877967 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.877903    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh" event={"ID":"4eb4bf49-c243-426b-a143-22c987695db4","Type":"ContainerStarted","Data":"c585b1883005a17184a95d7fe83ba2cf16d60967d295983e6ecac560a2efafbc"}
Sep 15 21:26:25.877967 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.877953    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh" event={"ID":"4eb4bf49-c243-426b-a143-22c987695db4","Type":"ContainerStarted","Data":"99c1317d2740d0673b64366d7e4aadc88ed4aa4e363ac2eac3fc7fc30c74b6d5"}
Sep 15 21:26:25.878082 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.877999    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh" event={"ID":"4eb4bf49-c243-426b-a143-22c987695db4","Type":"ContainerStarted","Data":"9e9741963668e10c637a9f414e0c819912d728f5f06407f1dcb6cdb6ce6658e2"}
Sep 15 21:26:25.878082 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.878045    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh" event={"ID":"4eb4bf49-c243-426b-a143-22c987695db4","Type":"ContainerStarted","Data":"ab181c16f7e0d404b0262e52a18d7cce741db6ee7c8adf018dae58a63537b74d"}
Sep 15 21:26:25.878082 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.878077    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh" event={"ID":"4eb4bf49-c243-426b-a143-22c987695db4","Type":"ContainerStarted","Data":"81fb624f990d4f1f9442d45a831eb1607a2cd7d77e78ebcaaf5f6f114f59a434"}
Sep 15 21:26:25.879446 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.879425    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wwhph" event={"ID":"6b3f47d0-ec5b-4699-91a9-160ea99f70fc","Type":"ContainerStarted","Data":"4faabd528c21b088a178070aa84aaeeccfb0b9e35e4a8164e0dee6105b69c09c"}
Sep 15 21:26:25.881105 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.881078    2575 generic.go:356] "Generic (PLEG): container finished" podID="2d221fc3-4447-44fe-bf1e-6bb0890ce88a" containerID="baad1c2b605de30a421703fef6afc3d7429897c5727d369b303b9e7764d097d9" exitCode=0
Sep 15 21:26:25.881212 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.881136    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmmgp" event={"ID":"2d221fc3-4447-44fe-bf1e-6bb0890ce88a","Type":"ContainerDied","Data":"baad1c2b605de30a421703fef6afc3d7429897c5727d369b303b9e7764d097d9"}
Sep 15 21:26:25.882939 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.882920    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rwmx9" event={"ID":"895233b3-517c-4b44-9dc1-02cb2e4ff8ac","Type":"ContainerStarted","Data":"2965799f2865b06ca23b4399217224f3187eee5d0fb86d6444b6f9868e3c9b7b"}
Sep 15 21:26:25.884812 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.884792    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zvtxn" event={"ID":"722562f8-7296-47bd-8fec-1d6791c5afd0","Type":"ContainerStarted","Data":"02fd83d830c3159ee23fd039b92c7b053dd5085d4c5ef4a9ddb7c675e778055c"}
Sep 15 21:26:25.886180 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.886140    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-js9vq" podStartSLOduration=3.0935790499999998 podStartE2EDuration="22.886129176s" totalImagesPullingTime="19.792550126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:25.886107018 +0000 UTC m=+23.698115473" watchObservedRunningTime="2026-09-15 21:26:25.886129176 +0000 UTC m=+23.698137623"
Sep 15 21:26:25.902635 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.902586    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zvtxn" podStartSLOduration=3.128156799 podStartE2EDuration="22.902570592s" totalImagesPullingTime="19.774413793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:25.902075675 +0000 UTC m=+23.714084120" watchObservedRunningTime="2026-09-15 21:26:25.902570592 +0000 UTC m=+23.714579036"
Sep 15 21:26:25.935058 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.935015    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rwmx9" podStartSLOduration=3.17600691 podStartE2EDuration="22.934998518s" totalImagesPullingTime="19.758991608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:25.914794306 +0000 UTC m=+23.726802750" watchObservedRunningTime="2026-09-15 21:26:25.934998518 +0000 UTC m=+23.747006964"
Sep 15 21:26:25.948278 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:25.948239    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wwhph" podStartSLOduration=3.170507355 podStartE2EDuration="22.948226084s" totalImagesPullingTime="19.777718729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:25.948066917 +0000 UTC m=+23.760075361" watchObservedRunningTime="2026-09-15 21:26:25.948226084 +0000 UTC m=+23.760234528"
Sep 15 21:26:26.362262 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:26.362074    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 21:26:26.667947 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:26.667803    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T21:26:26.362261456Z","UUID":"b1fc8481-fb22-424b-b989-7c1c1b2ccec1","Handler":null,"Name":"","Endpoint":""}
Sep 15 21:26:26.669465 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:26.669439    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 21:26:26.669465 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:26.669470    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 21:26:26.743700 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:26.743673    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:26.743860 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:26.743790    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:26.889272 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:26.889231    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf" event={"ID":"488a7507-e77a-4520-8855-4ec61e80c149","Type":"ContainerStarted","Data":"fe5910414f7366fce82a14ab221e037ae750e77cf26a92e11a04cbeb46bd79ae"}
Sep 15 21:26:26.890891 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:26.890861    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tjsqw" event={"ID":"83ec8a70-a84e-4da5-a0db-f51c03a2dac1","Type":"ContainerStarted","Data":"01155a29caf8f495c8915ba2ef6cf4bb9f2709d2500be699b7e265fea8b98df6"}
Sep 15 21:26:26.905200 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:26.905163    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tjsqw" podStartSLOduration=4.178251351 podStartE2EDuration="23.905149836s" totalImagesPullingTime="19.726898485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:26.904779705 +0000 UTC m=+24.716788150" watchObservedRunningTime="2026-09-15 21:26:26.905149836 +0000 UTC m=+24.717158279"
Sep 15 21:26:27.741401 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:27.741319    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:27.741559 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:27.741440    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:27.741863 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:27.741845    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:27.741966 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:27.741951    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:27.755795 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:27.755773    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zvtxn"
Sep 15 21:26:27.756282 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:27.756260    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zvtxn"
Sep 15 21:26:27.898166 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:27.898136    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh" event={"ID":"4eb4bf49-c243-426b-a143-22c987695db4","Type":"ContainerStarted","Data":"eef34ad83f184164f61f05ed3590abaf4e6a668708c4804b46568ad6f32304c9"}
Sep 15 21:26:27.898535 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:27.898216    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zvtxn"
Sep 15 21:26:27.898946 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:27.898928    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zvtxn"
Sep 15 21:26:28.741124 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:28.741075    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:28.741338 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:28.741222    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:28.901987 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:28.901954    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf" event={"ID":"488a7507-e77a-4520-8855-4ec61e80c149","Type":"ContainerStarted","Data":"98f6b23fbec135e1285c5003e6b3f8ce6c7cec29aa1a7e641bfbe5d2f4785c06"}
Sep 15 21:26:28.920425 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:28.920381    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8trf" podStartSLOduration=3.076557284 podStartE2EDuration="25.920365253s" totalImagesPullingTime="22.843807969s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:26:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:28.919980396 +0000 UTC m=+26.731988841" watchObservedRunningTime="2026-09-15 21:26:28.920365253 +0000 UTC m=+26.732373696"
Sep 15 21:26:29.741038 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:29.741002    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:29.741202 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:29.741136    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:29.741202 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:29.741155    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:29.741295 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:29.741270    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:30.741232 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:30.741059    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:30.741691 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:30.741316    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:30.907864 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:30.907830    2575 generic.go:356] "Generic (PLEG): container finished" podID="2d221fc3-4447-44fe-bf1e-6bb0890ce88a" containerID="b94e5be9761bbb1cb1f5f9a5b89986887fb34736b171ff22ddb795fee5d56626" exitCode=0
Sep 15 21:26:30.908005 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:30.907908    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmmgp" event={"ID":"2d221fc3-4447-44fe-bf1e-6bb0890ce88a","Type":"ContainerDied","Data":"b94e5be9761bbb1cb1f5f9a5b89986887fb34736b171ff22ddb795fee5d56626"}
Sep 15 21:26:30.911611 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:30.911591    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh" event={"ID":"4eb4bf49-c243-426b-a143-22c987695db4","Type":"ContainerStarted","Data":"ab5bb555560e9417e7e0bf756ef23679126dc5ecad399ee9d752ba278f3a8153"}
Sep 15 21:26:30.911985 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:30.911968    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:30.912086 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:30.912030    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:30.912086 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:30.912074    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:30.927280 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:30.927256    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:30.927411 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:30.927400    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:26:30.953652 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:30.953613    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh" podStartSLOduration=8.101613699 podStartE2EDuration="27.953601359s" totalImagesPullingTime="19.85198766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:30.952047692 +0000 UTC m=+28.764056136" watchObservedRunningTime="2026-09-15 21:26:30.953601359 +0000 UTC m=+28.765609781"
Sep 15 21:26:31.746438 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:31.741419    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:31.746438 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:31.741535    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:31.746438 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:31.741419    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:31.746438 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:31.741655    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:32.502101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:32.501864    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b46fs"]
Sep 15 21:26:32.502228 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:32.502106    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:32.502228 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:32.502187    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:32.504665 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:32.504640    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rsbdn"]
Sep 15 21:26:32.504797 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:32.504731    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:32.504876 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:32.504855    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:32.513048 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:32.513024    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v6j25"]
Sep 15 21:26:32.513122 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:32.513092    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:32.513190 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:32.513174    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:32.917853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:32.917816    2575 generic.go:356] "Generic (PLEG): container finished" podID="2d221fc3-4447-44fe-bf1e-6bb0890ce88a" containerID="90152aad121e0537e7bf1f6872bf71ec1ee15bbfa7837efd846febbabbb2404a" exitCode=0
Sep 15 21:26:32.918344 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:32.917898    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmmgp" event={"ID":"2d221fc3-4447-44fe-bf1e-6bb0890ce88a","Type":"ContainerDied","Data":"90152aad121e0537e7bf1f6872bf71ec1ee15bbfa7837efd846febbabbb2404a"}
Sep 15 21:26:33.541618 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:33.541576    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rwmx9_895233b3-517c-4b44-9dc1-02cb2e4ff8ac/dns-node-resolver/0.log"
Sep 15 21:26:33.741723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:33.741694    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:33.741908 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:33.741824    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:33.741908 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:33.741875    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:33.741990 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:33.741972    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:34.517689 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:34.517666    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wwhph_6b3f47d0-ec5b-4699-91a9-160ea99f70fc/node-ca/0.log"
Sep 15 21:26:34.741400 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:34.741355    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:34.741549 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:34.741473    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:34.923615 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:34.923584    2575 generic.go:356] "Generic (PLEG): container finished" podID="2d221fc3-4447-44fe-bf1e-6bb0890ce88a" containerID="20f48b9a28b331c0cd7a2051d13b8e915ee3a23172ba35592fce8e347fcd0268" exitCode=0
Sep 15 21:26:34.923792 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:34.923637    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmmgp" event={"ID":"2d221fc3-4447-44fe-bf1e-6bb0890ce88a","Type":"ContainerDied","Data":"20f48b9a28b331c0cd7a2051d13b8e915ee3a23172ba35592fce8e347fcd0268"}
Sep 15 21:26:35.741676 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:35.741641    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:35.742104 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:35.741678    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:35.742104 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:35.741774    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:35.742104 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:35.741918    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:36.361664 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:36.361630    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret\") pod \"global-pull-secret-syncer-v6j25\" (UID: \"21784d6c-c661-42eb-8b09-41376dacc1a3\") " pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:36.361885 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:36.361778    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 21:26:36.361885 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:36.361852    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret podName:21784d6c-c661-42eb-8b09-41376dacc1a3 nodeName:}" failed. No retries permitted until 2026-09-15 21:27:08.361838382 +0000 UTC m=+66.173846817 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret") pod "global-pull-secret-syncer-v6j25" (UID: "21784d6c-c661-42eb-8b09-41376dacc1a3") : object "kube-system"/"original-pull-secret" not registered
Sep 15 21:26:36.461927 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:36.461893    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs\") pod \"network-metrics-daemon-b46fs\" (UID: \"117891c4-3453-4091-81f0-46b5532f49b8\") " pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:36.462082 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:36.461957    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqn4c\" (UniqueName: \"kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c\") pod \"network-check-target-rsbdn\" (UID: \"f508cca8-4d27-4ee5-926d-1a7a3ce42286\") " pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:36.462082 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:36.462045    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:26:36.462082 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:36.462057    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 21:26:36.462082 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:36.462080    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 21:26:36.462207 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:36.462090    2575 projected.go:196] Error preparing data for projected volume kube-api-access-qqn4c for pod openshift-network-diagnostics/network-check-target-rsbdn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:26:36.462207 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:36.462106    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs podName:117891c4-3453-4091-81f0-46b5532f49b8 nodeName:}" failed. No retries permitted until 2026-09-15 21:27:08.462092123 +0000 UTC m=+66.274100549 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs") pod "network-metrics-daemon-b46fs" (UID: "117891c4-3453-4091-81f0-46b5532f49b8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 21:26:36.462207 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:36.462126    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c podName:f508cca8-4d27-4ee5-926d-1a7a3ce42286 nodeName:}" failed. No retries permitted until 2026-09-15 21:27:08.462114379 +0000 UTC m=+66.274122827 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qqn4c" (UniqueName: "kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c") pod "network-check-target-rsbdn" (UID: "f508cca8-4d27-4ee5-926d-1a7a3ce42286") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 21:26:36.741531 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:36.741464    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:36.741814 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:36.741579    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:37.741702 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:37.741672    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:37.741702 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:37.741696    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:37.742228 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:37.741801    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:37.742228 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:37.741950    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:38.741142 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:38.741106    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:38.741336 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:38.741246    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:39.741447 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:39.741403    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:39.741447 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:39.741445    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:39.741947 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:39.741543    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:39.741947 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:39.741677    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:40.741827 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:40.741570    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:40.742148 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:40.741853    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:40.942330 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:40.942302    2575 generic.go:356] "Generic (PLEG): container finished" podID="2d221fc3-4447-44fe-bf1e-6bb0890ce88a" containerID="d4709becc93de96ff0df55539a4388ec4052a5b690b37a0a78d6b0f435cd35c0" exitCode=0
Sep 15 21:26:40.942455 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:40.942340    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmmgp" event={"ID":"2d221fc3-4447-44fe-bf1e-6bb0890ce88a","Type":"ContainerDied","Data":"d4709becc93de96ff0df55539a4388ec4052a5b690b37a0a78d6b0f435cd35c0"}
Sep 15 21:26:41.741479 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:41.741450    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:41.741661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:41.741451    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:41.741661 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:41.741570    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:41.741661 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:41.741631    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:41.948246 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:41.948167    2575 generic.go:356] "Generic (PLEG): container finished" podID="2d221fc3-4447-44fe-bf1e-6bb0890ce88a" containerID="b02f716e46ff9be683ea6e5652883787e8883c0a5e7dc3d9d81ba132b1b31700" exitCode=0
Sep 15 21:26:41.948246 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:41.948214    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmmgp" event={"ID":"2d221fc3-4447-44fe-bf1e-6bb0890ce88a","Type":"ContainerDied","Data":"b02f716e46ff9be683ea6e5652883787e8883c0a5e7dc3d9d81ba132b1b31700"}
Sep 15 21:26:42.742543 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:42.742509    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:42.742707 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:42.742617    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:42.954078 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:42.954040    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jmmgp" event={"ID":"2d221fc3-4447-44fe-bf1e-6bb0890ce88a","Type":"ContainerStarted","Data":"d23cdf0a90bec07f70470901570615b504ef734836f9f512b97e0f4d98150433"}
Sep 15 21:26:42.977813 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:42.977769    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jmmgp" podStartSLOduration=7.676172135 podStartE2EDuration="39.977740901s" totalImagesPullingTime="32.301568766s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:26:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:26:42.977264122 +0000 UTC m=+40.789272565" watchObservedRunningTime="2026-09-15 21:26:42.977740901 +0000 UTC m=+40.789749344"
Sep 15 21:26:43.740666 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:43.740633    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:43.740842 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:43.740744    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:43.740842 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:43.740799    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:43.740920 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:43.740884    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:44.741570 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:44.741546    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:44.741970 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:44.741638    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:45.741683 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:45.741652    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:45.742108 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:45.741775    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:45.742108 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:45.741819    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:45.742636 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:45.742601    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:46.741064 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:46.741035    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:46.741238 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:46.741132    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:47.740958 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:47.740926    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:47.741360 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:47.740940    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:47.741360 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:47.741019    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:47.741360 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:47.741094    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:48.740879 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:48.740844    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:48.741119 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:48.740971    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:49.741364 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:49.741327    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:49.741797 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:49.741341    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:49.741797 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:49.741442    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:49.741797 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:49.741535    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:50.740884 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:50.740857    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:50.741045 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:50.740942    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:51.741177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:51.741144    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:51.741721 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:51.741252    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:51.741721 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:51.741305    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:51.741721 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:51.741382    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:52.742404 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:52.742377    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:52.742749 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:52.742450    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:53.740963 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:53.740930    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:53.741131 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:53.740971    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:53.741131 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:53.741033    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:53.741204 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:53.741154    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:54.741633 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:54.741598    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:54.742154 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:54.741710    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:55.741701 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:55.741667    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:55.742110 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:55.741791    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:55.742110 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:55.741840    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:55.742110 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:55.741926    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:56.741037 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:56.741007    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:56.741221 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:56.741106    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rsbdn" podUID="f508cca8-4d27-4ee5-926d-1a7a3ce42286"
Sep 15 21:26:57.740646 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:57.740618    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:57.741048 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:57.740711    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-b46fs" podUID="117891c4-3453-4091-81f0-46b5532f49b8"
Sep 15 21:26:57.741048 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:57.740787    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:57.741048 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:26:57.740857    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v6j25" podUID="21784d6c-c661-42eb-8b09-41376dacc1a3"
Sep 15 21:26:58.449169 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.449138    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-148.ec2.internal" event="NodeReady"
Sep 15 21:26:58.449320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.449247    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 21:26:58.500750 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.500724    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sqhn2"]
Sep 15 21:26:58.502372 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.502357    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sqhn2"
Sep 15 21:26:58.503539 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.503519    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bqkb9"]
Sep 15 21:26:58.506408 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.505553    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.506408 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.505637    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 21:26:58.506408 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.505799    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 21:26:58.506408 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.505559    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7gcfd\""
Sep 15 21:26:58.508411 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.508390    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 21:26:58.508911 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.508711    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 21:26:58.508911 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.508716    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fd7sr\""
Sep 15 21:26:58.508911 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.508813    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 21:26:58.508911 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.508858    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 21:26:58.522372 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.522328    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bqkb9"]
Sep 15 21:26:58.522942 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.522925    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sqhn2"]
Sep 15 21:26:58.599275 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.599245    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-k2lzl"]
Sep 15 21:26:58.600880 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.600852    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k2lzl"
Sep 15 21:26:58.604404 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.604385    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 21:26:58.604608 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.604484    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 21:26:58.605674 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.605651    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-646rb\""
Sep 15 21:26:58.608355 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.608337    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 21:26:58.608444 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.608370    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2de7b26-21f1-475f-b5e4-fe26b4032240-data-volume\") pod \"insights-runtime-extractor-bqkb9\" (UID: \"b2de7b26-21f1-475f-b5e4-fe26b4032240\") " pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.608444 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.608403    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe46385-fe7f-418f-abd9-107a9da93dba-config-volume\") pod \"dns-default-sqhn2\" (UID: \"dbe46385-fe7f-418f-abd9-107a9da93dba\") " pod="openshift-dns/dns-default-sqhn2"
Sep 15 21:26:58.608444 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.608431    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe46385-fe7f-418f-abd9-107a9da93dba-metrics-tls\") pod \"dns-default-sqhn2\" (UID: \"dbe46385-fe7f-418f-abd9-107a9da93dba\") " pod="openshift-dns/dns-default-sqhn2"
Sep 15 21:26:58.608616 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.608468    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2de7b26-21f1-475f-b5e4-fe26b4032240-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bqkb9\" (UID: \"b2de7b26-21f1-475f-b5e4-fe26b4032240\") " pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.608616 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.608506    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hp9t\" (UniqueName: \"kubernetes.io/projected/dbe46385-fe7f-418f-abd9-107a9da93dba-kube-api-access-5hp9t\") pod \"dns-default-sqhn2\" (UID: \"dbe46385-fe7f-418f-abd9-107a9da93dba\") " pod="openshift-dns/dns-default-sqhn2"
Sep 15 21:26:58.608616 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.608528    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe46385-fe7f-418f-abd9-107a9da93dba-tmp-dir\") pod \"dns-default-sqhn2\" (UID: \"dbe46385-fe7f-418f-abd9-107a9da93dba\") " pod="openshift-dns/dns-default-sqhn2"
Sep 15 21:26:58.608616 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.608581    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpkcs\" (UniqueName: \"kubernetes.io/projected/b2de7b26-21f1-475f-b5e4-fe26b4032240-kube-api-access-jpkcs\") pod \"insights-runtime-extractor-bqkb9\" (UID: \"b2de7b26-21f1-475f-b5e4-fe26b4032240\") " pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.608616 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.608616    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2de7b26-21f1-475f-b5e4-fe26b4032240-crio-socket\") pod \"insights-runtime-extractor-bqkb9\" (UID: \"b2de7b26-21f1-475f-b5e4-fe26b4032240\") " pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.608861 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.608633    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2de7b26-21f1-475f-b5e4-fe26b4032240-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bqkb9\" (UID: \"b2de7b26-21f1-475f-b5e4-fe26b4032240\") " pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.693932 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.693895    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k2lzl"]
Sep 15 21:26:58.709319 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.709299    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpkcs\" (UniqueName: \"kubernetes.io/projected/b2de7b26-21f1-475f-b5e4-fe26b4032240-kube-api-access-jpkcs\") pod \"insights-runtime-extractor-bqkb9\" (UID: \"b2de7b26-21f1-475f-b5e4-fe26b4032240\") " pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.709421 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.709327    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2de7b26-21f1-475f-b5e4-fe26b4032240-crio-socket\") pod \"insights-runtime-extractor-bqkb9\" (UID: \"b2de7b26-21f1-475f-b5e4-fe26b4032240\") " pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.709421 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.709344    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2de7b26-21f1-475f-b5e4-fe26b4032240-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bqkb9\" (UID: \"b2de7b26-21f1-475f-b5e4-fe26b4032240\") " pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.709421 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.709366    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2de7b26-21f1-475f-b5e4-fe26b4032240-data-volume\") pod \"insights-runtime-extractor-bqkb9\" (UID: \"b2de7b26-21f1-475f-b5e4-fe26b4032240\") " pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.709421 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.709391    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe46385-fe7f-418f-abd9-107a9da93dba-config-volume\") pod \"dns-default-sqhn2\" (UID: \"dbe46385-fe7f-418f-abd9-107a9da93dba\") " pod="openshift-dns/dns-default-sqhn2"
Sep 15 21:26:58.709421 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.709412    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe46385-fe7f-418f-abd9-107a9da93dba-metrics-tls\") pod \"dns-default-sqhn2\" (UID: \"dbe46385-fe7f-418f-abd9-107a9da93dba\") " pod="openshift-dns/dns-default-sqhn2"
Sep 15 21:26:58.709659 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.709439    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2de7b26-21f1-475f-b5e4-fe26b4032240-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bqkb9\" (UID: \"b2de7b26-21f1-475f-b5e4-fe26b4032240\") " pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.709659 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.709555    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b2de7b26-21f1-475f-b5e4-fe26b4032240-crio-socket\") pod \"insights-runtime-extractor-bqkb9\" (UID: \"b2de7b26-21f1-475f-b5e4-fe26b4032240\") " pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.709659 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.709574    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hp9t\" (UniqueName: \"kubernetes.io/projected/dbe46385-fe7f-418f-abd9-107a9da93dba-kube-api-access-5hp9t\") pod \"dns-default-sqhn2\" (UID: \"dbe46385-fe7f-418f-abd9-107a9da93dba\") " pod="openshift-dns/dns-default-sqhn2"
Sep 15 21:26:58.709659 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.709614    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18d9cbb1-4fad-45f1-ae92-4125ca00a324-cert\") pod \"ingress-canary-k2lzl\" (UID: \"18d9cbb1-4fad-45f1-ae92-4125ca00a324\") " pod="openshift-ingress-canary/ingress-canary-k2lzl"
Sep 15 21:26:58.709659 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.709650    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe46385-fe7f-418f-abd9-107a9da93dba-tmp-dir\") pod \"dns-default-sqhn2\" (UID: \"dbe46385-fe7f-418f-abd9-107a9da93dba\") " pod="openshift-dns/dns-default-sqhn2"
Sep 15 21:26:58.709884 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.709675    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx88k\" (UniqueName: \"kubernetes.io/projected/18d9cbb1-4fad-45f1-ae92-4125ca00a324-kube-api-access-mx88k\") pod \"ingress-canary-k2lzl\" (UID: \"18d9cbb1-4fad-45f1-ae92-4125ca00a324\") " pod="openshift-ingress-canary/ingress-canary-k2lzl"
Sep 15 21:26:58.709884 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.709815    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2de7b26-21f1-475f-b5e4-fe26b4032240-data-volume\") pod \"insights-runtime-extractor-bqkb9\" (UID: \"b2de7b26-21f1-475f-b5e4-fe26b4032240\") " pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.710044 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.710022    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dbe46385-fe7f-418f-abd9-107a9da93dba-config-volume\") pod \"dns-default-sqhn2\" (UID: \"dbe46385-fe7f-418f-abd9-107a9da93dba\") " pod="openshift-dns/dns-default-sqhn2"
Sep 15 21:26:58.710097 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.710076    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbe46385-fe7f-418f-abd9-107a9da93dba-tmp-dir\") pod \"dns-default-sqhn2\" (UID: \"dbe46385-fe7f-418f-abd9-107a9da93dba\") " pod="openshift-dns/dns-default-sqhn2"
Sep 15 21:26:58.710146 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.710103    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b2de7b26-21f1-475f-b5e4-fe26b4032240-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bqkb9\" (UID: \"b2de7b26-21f1-475f-b5e4-fe26b4032240\") " pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.713382 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.713365    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b2de7b26-21f1-475f-b5e4-fe26b4032240-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bqkb9\" (UID: \"b2de7b26-21f1-475f-b5e4-fe26b4032240\") " pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.713382 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.713374    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbe46385-fe7f-418f-abd9-107a9da93dba-metrics-tls\") pod \"dns-default-sqhn2\" (UID: \"dbe46385-fe7f-418f-abd9-107a9da93dba\") " pod="openshift-dns/dns-default-sqhn2"
Sep 15 21:26:58.718447 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.718416    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hp9t\" (UniqueName: \"kubernetes.io/projected/dbe46385-fe7f-418f-abd9-107a9da93dba-kube-api-access-5hp9t\") pod \"dns-default-sqhn2\" (UID: \"dbe46385-fe7f-418f-abd9-107a9da93dba\") " pod="openshift-dns/dns-default-sqhn2"
Sep 15 21:26:58.718730 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.718711    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpkcs\" (UniqueName: \"kubernetes.io/projected/b2de7b26-21f1-475f-b5e4-fe26b4032240-kube-api-access-jpkcs\") pod \"insights-runtime-extractor-bqkb9\" (UID: \"b2de7b26-21f1-475f-b5e4-fe26b4032240\") " pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.740664 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.740642    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:26:58.743007 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.742990    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pqjdv\""
Sep 15 21:26:58.743103 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.742990    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 21:26:58.743103 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.743048    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 21:26:58.810383 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.810332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18d9cbb1-4fad-45f1-ae92-4125ca00a324-cert\") pod \"ingress-canary-k2lzl\" (UID: \"18d9cbb1-4fad-45f1-ae92-4125ca00a324\") " pod="openshift-ingress-canary/ingress-canary-k2lzl"
Sep 15 21:26:58.810383 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.810358    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx88k\" (UniqueName: \"kubernetes.io/projected/18d9cbb1-4fad-45f1-ae92-4125ca00a324-kube-api-access-mx88k\") pod \"ingress-canary-k2lzl\" (UID: \"18d9cbb1-4fad-45f1-ae92-4125ca00a324\") " pod="openshift-ingress-canary/ingress-canary-k2lzl"
Sep 15 21:26:58.812603 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.812580    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/18d9cbb1-4fad-45f1-ae92-4125ca00a324-cert\") pod \"ingress-canary-k2lzl\" (UID: \"18d9cbb1-4fad-45f1-ae92-4125ca00a324\") " pod="openshift-ingress-canary/ingress-canary-k2lzl"
Sep 15 21:26:58.814445 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.814431    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sqhn2"
Sep 15 21:26:58.818192 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.818170    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx88k\" (UniqueName: \"kubernetes.io/projected/18d9cbb1-4fad-45f1-ae92-4125ca00a324-kube-api-access-mx88k\") pod \"ingress-canary-k2lzl\" (UID: \"18d9cbb1-4fad-45f1-ae92-4125ca00a324\") " pod="openshift-ingress-canary/ingress-canary-k2lzl"
Sep 15 21:26:58.818355 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.818343    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bqkb9"
Sep 15 21:26:58.909127 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:58.909103    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k2lzl"
Sep 15 21:26:59.092380 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:59.092314    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sqhn2"]
Sep 15 21:26:59.095687 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:59.095660    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bqkb9"]
Sep 15 21:26:59.098606 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:59.098587    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2de7b26_21f1_475f_b5e4_fe26b4032240.slice/crio-b2fc0d69f7f9a295bc3ffdcc21d3bdd419c245671bf25e293051957bb720718f WatchSource:0}: Error finding container b2fc0d69f7f9a295bc3ffdcc21d3bdd419c245671bf25e293051957bb720718f: Status 404 returned error can't find the container with id b2fc0d69f7f9a295bc3ffdcc21d3bdd419c245671bf25e293051957bb720718f
Sep 15 21:26:59.172776 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:59.172730    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k2lzl"]
Sep 15 21:26:59.175468 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:26:59.175439    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d9cbb1_4fad_45f1_ae92_4125ca00a324.slice/crio-854283cd715ea415527f71440259b178ee3663cb8d1943af0bbe96bd38df0bae WatchSource:0}: Error finding container 854283cd715ea415527f71440259b178ee3663cb8d1943af0bbe96bd38df0bae: Status 404 returned error can't find the container with id 854283cd715ea415527f71440259b178ee3663cb8d1943af0bbe96bd38df0bae
Sep 15 21:26:59.740840 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:59.740811    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:26:59.740840 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:59.740825    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:26:59.747777 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:59.744323    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 21:26:59.747777 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:59.744400    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 21:26:59.747777 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:59.744426    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q9hj2\""
Sep 15 21:26:59.992849 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:59.992746    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k2lzl" event={"ID":"18d9cbb1-4fad-45f1-ae92-4125ca00a324","Type":"ContainerStarted","Data":"854283cd715ea415527f71440259b178ee3663cb8d1943af0bbe96bd38df0bae"}
Sep 15 21:26:59.995259 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:59.995118    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bqkb9" event={"ID":"b2de7b26-21f1-475f-b5e4-fe26b4032240","Type":"ContainerStarted","Data":"c94d7c61c475c7cc0a0f8f6d536d56729ddedad832011b4ef307522745ed089e"}
Sep 15 21:26:59.995259 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:59.995177    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bqkb9" event={"ID":"b2de7b26-21f1-475f-b5e4-fe26b4032240","Type":"ContainerStarted","Data":"0f8b5f1852779bfec80063a4c42b1a2f1a339736f59f334d9a3fe7eb24a92a9d"}
Sep 15 21:26:59.995259 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:59.995210    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bqkb9" event={"ID":"b2de7b26-21f1-475f-b5e4-fe26b4032240","Type":"ContainerStarted","Data":"b2fc0d69f7f9a295bc3ffdcc21d3bdd419c245671bf25e293051957bb720718f"}
Sep 15 21:26:59.996628 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:26:59.996605    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sqhn2" event={"ID":"dbe46385-fe7f-418f-abd9-107a9da93dba","Type":"ContainerStarted","Data":"c487cd0c72f60ec82516f5341c677c500c7f0be32a9a767bdf3b954e9dcd7e33"}
Sep 15 21:27:01.001510 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:01.001418    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sqhn2" event={"ID":"dbe46385-fe7f-418f-abd9-107a9da93dba","Type":"ContainerStarted","Data":"208ff0fb96186c8ee3135c9f0509f061c8a3aebb87ef64555a6246a1fb0ca74e"}
Sep 15 21:27:01.001510 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:01.001478    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sqhn2" event={"ID":"dbe46385-fe7f-418f-abd9-107a9da93dba","Type":"ContainerStarted","Data":"4376a342504823295dc6dd4b254d3b010e0600893469dfba04b5c4e9582cddc4"}
Sep 15 21:27:01.001947 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:01.001527    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sqhn2"
Sep 15 21:27:01.022710 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:01.020395    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sqhn2" podStartSLOduration=1.429326618 podStartE2EDuration="3.020381539s" totalImagesPullingTime="1.591054921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:01.01970415 +0000 UTC m=+58.831712594" watchObservedRunningTime="2026-09-15 21:27:01.020381539 +0000 UTC m=+58.832390027"
Sep 15 21:27:02.934682 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:02.934654    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mpwmh"
Sep 15 21:27:03.008197 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.008168    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k2lzl" event={"ID":"18d9cbb1-4fad-45f1-ae92-4125ca00a324","Type":"ContainerStarted","Data":"645a02529f1ab4a37580d275608adf39730297324a6ee2dd353b4aa9066d3dcc"}
Sep 15 21:27:03.010212 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.010191    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bqkb9" event={"ID":"b2de7b26-21f1-475f-b5e4-fe26b4032240","Type":"ContainerStarted","Data":"9d96cf3d33fdf813d929e0880f431d509dcab15ea89038e072066b5e23f5a762"}
Sep 15 21:27:03.054318 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.054277    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-k2lzl" podStartSLOduration=2.069670005 podStartE2EDuration="5.054263187s" totalImagesPullingTime="2.984593182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:03.052702487 +0000 UTC m=+60.864710929" watchObservedRunningTime="2026-09-15 21:27:03.054263187 +0000 UTC m=+60.866271641"
Sep 15 21:27:03.077209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.077155    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bqkb9" podStartSLOduration=2.137876227 podStartE2EDuration="5.077134791s" totalImagesPullingTime="2.939258564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:26:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:03.076212096 +0000 UTC m=+60.888220577" watchObservedRunningTime="2026-09-15 21:27:03.077134791 +0000 UTC m=+60.889143235"
Sep 15 21:27:03.509163 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.509125    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8dcdf8955-425x9"]
Sep 15 21:27:03.525306 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.525270    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8dcdf8955-425x9"]
Sep 15 21:27:03.525444 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.525359    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.527663 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.527641    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 21:27:03.527902 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.527878    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-hqrg5\""
Sep 15 21:27:03.527902 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.527898    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 15 21:27:03.528785 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.528740    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 21:27:03.528865 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.528749    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 15 21:27:03.528865 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.528745    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 15 21:27:03.528865 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.528745    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 15 21:27:03.528865 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.528815    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 15 21:27:03.547236 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.547219    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-oauth-config\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.547336 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.547243    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-config\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.547336 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.547261    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-oauth-serving-cert\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.547336 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.547293    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-service-ca\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.547438 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.547355    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-serving-cert\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.547438 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.547386    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw5lh\" (UniqueName: \"kubernetes.io/projected/b95a4f87-5e3b-47ce-85e1-7e5513324690-kube-api-access-sw5lh\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.647682 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.647648    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-serving-cert\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.647682 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.647686    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw5lh\" (UniqueName: \"kubernetes.io/projected/b95a4f87-5e3b-47ce-85e1-7e5513324690-kube-api-access-sw5lh\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.647917 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.647734    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-oauth-config\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.647917 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.647787    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-config\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.647917 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.647809    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-oauth-serving-cert\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.647917 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.647856    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-service-ca\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.648605 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.648582    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-service-ca\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.648708 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.648583    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-oauth-serving-cert\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.648796 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.648777    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-config\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.650233 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.650204    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-serving-cert\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.650319 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.650252    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-oauth-config\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.656481 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.656460    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw5lh\" (UniqueName: \"kubernetes.io/projected/b95a4f87-5e3b-47ce-85e1-7e5513324690-kube-api-access-sw5lh\") pod \"console-8dcdf8955-425x9\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") " pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:03.834559 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:03.834472    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:04.099672 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:04.099646    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8dcdf8955-425x9"]
Sep 15 21:27:04.102820 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:27:04.102799    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95a4f87_5e3b_47ce_85e1_7e5513324690.slice/crio-4ff46e4088741d3cc90b81ede92ff274cdfde008a3dfaa088d772b8f2cad0651 WatchSource:0}: Error finding container 4ff46e4088741d3cc90b81ede92ff274cdfde008a3dfaa088d772b8f2cad0651: Status 404 returned error can't find the container with id 4ff46e4088741d3cc90b81ede92ff274cdfde008a3dfaa088d772b8f2cad0651
Sep 15 21:27:05.018272 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.018235    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dcdf8955-425x9" event={"ID":"b95a4f87-5e3b-47ce-85e1-7e5513324690","Type":"ContainerStarted","Data":"4ff46e4088741d3cc90b81ede92ff274cdfde008a3dfaa088d772b8f2cad0651"}
Sep 15 21:27:05.300283 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.300205    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"]
Sep 15 21:27:05.302137 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.302117    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"
Sep 15 21:27:05.305744 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.305720    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 15 21:27:05.306044 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.306029    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vj4tb\""
Sep 15 21:27:05.307739 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.307691    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 15 21:27:05.307859 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.307796    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 21:27:05.307859 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.307728    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 21:27:05.307969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.307728    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 21:27:05.315263 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.315243    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"]
Sep 15 21:27:05.360509 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.360473    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/282361ed-7a1b-459a-adc2-276fdc8688a2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-5bgzn\" (UID: \"282361ed-7a1b-459a-adc2-276fdc8688a2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"
Sep 15 21:27:05.360667 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.360516    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdctd\" (UniqueName: \"kubernetes.io/projected/282361ed-7a1b-459a-adc2-276fdc8688a2-kube-api-access-kdctd\") pod \"prometheus-operator-68bcd7f987-5bgzn\" (UID: \"282361ed-7a1b-459a-adc2-276fdc8688a2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"
Sep 15 21:27:05.360667 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.360574    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/282361ed-7a1b-459a-adc2-276fdc8688a2-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-5bgzn\" (UID: \"282361ed-7a1b-459a-adc2-276fdc8688a2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"
Sep 15 21:27:05.360667 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.360602    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/282361ed-7a1b-459a-adc2-276fdc8688a2-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-5bgzn\" (UID: \"282361ed-7a1b-459a-adc2-276fdc8688a2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"
Sep 15 21:27:05.461223 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.461190    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/282361ed-7a1b-459a-adc2-276fdc8688a2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-5bgzn\" (UID: \"282361ed-7a1b-459a-adc2-276fdc8688a2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"
Sep 15 21:27:05.461400 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.461233    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdctd\" (UniqueName: \"kubernetes.io/projected/282361ed-7a1b-459a-adc2-276fdc8688a2-kube-api-access-kdctd\") pod \"prometheus-operator-68bcd7f987-5bgzn\" (UID: \"282361ed-7a1b-459a-adc2-276fdc8688a2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"
Sep 15 21:27:05.461571 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.461549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/282361ed-7a1b-459a-adc2-276fdc8688a2-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-5bgzn\" (UID: \"282361ed-7a1b-459a-adc2-276fdc8688a2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"
Sep 15 21:27:05.461643 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.461595    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/282361ed-7a1b-459a-adc2-276fdc8688a2-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-5bgzn\" (UID: \"282361ed-7a1b-459a-adc2-276fdc8688a2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"
Sep 15 21:27:05.461719 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:27:05.461702    2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 15 21:27:05.461814 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:27:05.461798    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/282361ed-7a1b-459a-adc2-276fdc8688a2-prometheus-operator-tls podName:282361ed-7a1b-459a-adc2-276fdc8688a2 nodeName:}" failed. No retries permitted until 2026-09-15 21:27:05.961779458 +0000 UTC m=+63.773787884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/282361ed-7a1b-459a-adc2-276fdc8688a2-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-5bgzn" (UID: "282361ed-7a1b-459a-adc2-276fdc8688a2") : secret "prometheus-operator-tls" not found
Sep 15 21:27:05.462323 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.462301    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/282361ed-7a1b-459a-adc2-276fdc8688a2-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-5bgzn\" (UID: \"282361ed-7a1b-459a-adc2-276fdc8688a2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"
Sep 15 21:27:05.464024 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.464002    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/282361ed-7a1b-459a-adc2-276fdc8688a2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-5bgzn\" (UID: \"282361ed-7a1b-459a-adc2-276fdc8688a2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"
Sep 15 21:27:05.472715 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.472694    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdctd\" (UniqueName: \"kubernetes.io/projected/282361ed-7a1b-459a-adc2-276fdc8688a2-kube-api-access-kdctd\") pod \"prometheus-operator-68bcd7f987-5bgzn\" (UID: \"282361ed-7a1b-459a-adc2-276fdc8688a2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"
Sep 15 21:27:05.964807 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:05.964773    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/282361ed-7a1b-459a-adc2-276fdc8688a2-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-5bgzn\" (UID: \"282361ed-7a1b-459a-adc2-276fdc8688a2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"
Sep 15 21:27:05.964970 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:27:05.964902    2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 15 21:27:05.965027 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:27:05.964993    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/282361ed-7a1b-459a-adc2-276fdc8688a2-prometheus-operator-tls podName:282361ed-7a1b-459a-adc2-276fdc8688a2 nodeName:}" failed. No retries permitted until 2026-09-15 21:27:06.96497218 +0000 UTC m=+64.776980601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/282361ed-7a1b-459a-adc2-276fdc8688a2-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-5bgzn" (UID: "282361ed-7a1b-459a-adc2-276fdc8688a2") : secret "prometheus-operator-tls" not found
Sep 15 21:27:06.973846 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:06.973811    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/282361ed-7a1b-459a-adc2-276fdc8688a2-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-5bgzn\" (UID: \"282361ed-7a1b-459a-adc2-276fdc8688a2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"
Sep 15 21:27:06.976174 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:06.976152    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/282361ed-7a1b-459a-adc2-276fdc8688a2-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-5bgzn\" (UID: \"282361ed-7a1b-459a-adc2-276fdc8688a2\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"
Sep 15 21:27:07.113182 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:07.113146    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"
Sep 15 21:27:07.383110 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:07.383077    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn"]
Sep 15 21:27:07.386668 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:27:07.386641    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod282361ed_7a1b_459a_adc2_276fdc8688a2.slice/crio-8c35cd14a62367eb4aa9362ff1f2eed46d1d6a6f7f5d9f1b9f25e404d1a84575 WatchSource:0}: Error finding container 8c35cd14a62367eb4aa9362ff1f2eed46d1d6a6f7f5d9f1b9f25e404d1a84575: Status 404 returned error can't find the container with id 8c35cd14a62367eb4aa9362ff1f2eed46d1d6a6f7f5d9f1b9f25e404d1a84575
Sep 15 21:27:08.028745 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.028705    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dcdf8955-425x9" event={"ID":"b95a4f87-5e3b-47ce-85e1-7e5513324690","Type":"ContainerStarted","Data":"b4fd7b4be0cb492ee8a69cfa02dbe39ecd0d2a171f11c38ded1bd6ba48f092d1"}
Sep 15 21:27:08.030112 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.030076    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn" event={"ID":"282361ed-7a1b-459a-adc2-276fdc8688a2","Type":"ContainerStarted","Data":"8c35cd14a62367eb4aa9362ff1f2eed46d1d6a6f7f5d9f1b9f25e404d1a84575"}
Sep 15 21:27:08.047642 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.047597    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8dcdf8955-425x9" podStartSLOduration=2.135413007 podStartE2EDuration="5.047583293s" totalImagesPullingTime="2.912170286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:08.046519839 +0000 UTC m=+65.858528296" watchObservedRunningTime="2026-09-15 21:27:08.047583293 +0000 UTC m=+65.859591736"
Sep 15 21:27:08.385116 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.384954    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret\") pod \"global-pull-secret-syncer-v6j25\" (UID: \"21784d6c-c661-42eb-8b09-41376dacc1a3\") " pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:27:08.387299 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.387279    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 21:27:08.398529 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.398503    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/21784d6c-c661-42eb-8b09-41376dacc1a3-original-pull-secret\") pod \"global-pull-secret-syncer-v6j25\" (UID: \"21784d6c-c661-42eb-8b09-41376dacc1a3\") " pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:27:08.453554 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.453516    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v6j25"
Sep 15 21:27:08.485412 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.485377    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqn4c\" (UniqueName: \"kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c\") pod \"network-check-target-rsbdn\" (UID: \"f508cca8-4d27-4ee5-926d-1a7a3ce42286\") " pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:27:08.485544 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.485432    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs\") pod \"network-metrics-daemon-b46fs\" (UID: \"117891c4-3453-4091-81f0-46b5532f49b8\") " pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:27:08.487909 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.487886    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 21:27:08.488030 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.487890    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 21:27:08.498422 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.498399    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 21:27:08.498963 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.498939    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/117891c4-3453-4091-81f0-46b5532f49b8-metrics-certs\") pod \"network-metrics-daemon-b46fs\" (UID: \"117891c4-3453-4091-81f0-46b5532f49b8\") " pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:27:08.509810 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.509788    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqn4c\" (UniqueName: \"kubernetes.io/projected/f508cca8-4d27-4ee5-926d-1a7a3ce42286-kube-api-access-qqn4c\") pod \"network-check-target-rsbdn\" (UID: \"f508cca8-4d27-4ee5-926d-1a7a3ce42286\") " pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:27:08.650664 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.650642    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pqjdv\""
Sep 15 21:27:08.659298 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.659267    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:27:08.761041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.761020    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-q9hj2\""
Sep 15 21:27:08.769408 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.769386    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-b46fs"
Sep 15 21:27:08.888411 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.888382    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v6j25"]
Sep 15 21:27:08.891419 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:27:08.891393    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21784d6c_c661_42eb_8b09_41376dacc1a3.slice/crio-11daf96f129a92edb96a6912805aca759a77a625683895ff057238ce9f8d93c3 WatchSource:0}: Error finding container 11daf96f129a92edb96a6912805aca759a77a625683895ff057238ce9f8d93c3: Status 404 returned error can't find the container with id 11daf96f129a92edb96a6912805aca759a77a625683895ff057238ce9f8d93c3
Sep 15 21:27:08.934258 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:08.934191    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rsbdn"]
Sep 15 21:27:08.937067 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:27:08.937042    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf508cca8_4d27_4ee5_926d_1a7a3ce42286.slice/crio-b11b7248a7b2fbc07459a4807585c74ff8eaabee3e8fd4b998e434d4b6388fdb WatchSource:0}: Error finding container b11b7248a7b2fbc07459a4807585c74ff8eaabee3e8fd4b998e434d4b6388fdb: Status 404 returned error can't find the container with id b11b7248a7b2fbc07459a4807585c74ff8eaabee3e8fd4b998e434d4b6388fdb
Sep 15 21:27:09.035650 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:09.035622    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-b46fs"]
Sep 15 21:27:09.036102 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:09.035869    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn" event={"ID":"282361ed-7a1b-459a-adc2-276fdc8688a2","Type":"ContainerStarted","Data":"2968b43d9f3c651aa3d436680d25594833a12eada898d1eac3f9c97dbb479b06"}
Sep 15 21:27:09.036102 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:09.035928    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn" event={"ID":"282361ed-7a1b-459a-adc2-276fdc8688a2","Type":"ContainerStarted","Data":"262ed4a56f8a77dc52c206fb8ef6627f6abf5ee3ff4b67b260459457e1a32cc0"}
Sep 15 21:27:09.037228 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:09.037210    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rsbdn" event={"ID":"f508cca8-4d27-4ee5-926d-1a7a3ce42286","Type":"ContainerStarted","Data":"b11b7248a7b2fbc07459a4807585c74ff8eaabee3e8fd4b998e434d4b6388fdb"}
Sep 15 21:27:09.038358 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:09.038332    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v6j25" event={"ID":"21784d6c-c661-42eb-8b09-41376dacc1a3","Type":"ContainerStarted","Data":"11daf96f129a92edb96a6912805aca759a77a625683895ff057238ce9f8d93c3"}
Sep 15 21:27:09.039029 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:27:09.039010    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117891c4_3453_4091_81f0_46b5532f49b8.slice/crio-ac7e32ef69af3c26b64bb210173a94ac2eb43ff2abd5c32f5203a412be1a3660 WatchSource:0}: Error finding container ac7e32ef69af3c26b64bb210173a94ac2eb43ff2abd5c32f5203a412be1a3660: Status 404 returned error can't find the container with id ac7e32ef69af3c26b64bb210173a94ac2eb43ff2abd5c32f5203a412be1a3660
Sep 15 21:27:09.052472 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:09.052432    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-5bgzn" podStartSLOduration=2.7932266390000002 podStartE2EDuration="4.052418999s" totalImagesPullingTime="1.25919236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:09.051735707 +0000 UTC m=+66.863744160" watchObservedRunningTime="2026-09-15 21:27:09.052418999 +0000 UTC m=+66.864427442"
Sep 15 21:27:10.046264 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.046225    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b46fs" event={"ID":"117891c4-3453-4091-81f0-46b5532f49b8","Type":"ContainerStarted","Data":"ac7e32ef69af3c26b64bb210173a94ac2eb43ff2abd5c32f5203a412be1a3660"}
Sep 15 21:27:10.670965 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.670920    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"]
Sep 15 21:27:10.673311 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.673287    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"
Sep 15 21:27:10.675932 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.675904    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 15 21:27:10.676052 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.675904    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 15 21:27:10.676888 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.676655    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-lbtjm\""
Sep 15 21:27:10.690512 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.690482    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"]
Sep 15 21:27:10.700159 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.699406    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"]
Sep 15 21:27:10.701484 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.701466    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.702526 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.702282    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ca90f7-2961-4e2d-9345-ec5585c65079-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-2ws2m\" (UID: \"e9ca90f7-2961-4e2d-9345-ec5585c65079\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"
Sep 15 21:27:10.702526 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.702393    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9ca90f7-2961-4e2d-9345-ec5585c65079-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-2ws2m\" (UID: \"e9ca90f7-2961-4e2d-9345-ec5585c65079\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"
Sep 15 21:27:10.702526 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.702427    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2d7r\" (UniqueName: \"kubernetes.io/projected/e9ca90f7-2961-4e2d-9345-ec5585c65079-kube-api-access-j2d7r\") pod \"openshift-state-metrics-67dcf8c979-2ws2m\" (UID: \"e9ca90f7-2961-4e2d-9345-ec5585c65079\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"
Sep 15 21:27:10.702526 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.702454    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9ca90f7-2961-4e2d-9345-ec5585c65079-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-2ws2m\" (UID: \"e9ca90f7-2961-4e2d-9345-ec5585c65079\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"
Sep 15 21:27:10.705259 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.705233    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q7wqq"]
Sep 15 21:27:10.706378 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.706356    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 15 21:27:10.706579 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.706563    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 15 21:27:10.706786 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.706771    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 15 21:27:10.707019 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.706999    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hmvx5\""
Sep 15 21:27:10.707983 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.707965    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.710230 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.710042    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 21:27:10.710230 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.710163    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9lv77\""
Sep 15 21:27:10.710369 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.710339    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 21:27:10.712925 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.712904    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 21:27:10.714093 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.714074    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"]
Sep 15 21:27:10.803669 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.803635    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mgzp\" (UniqueName: \"kubernetes.io/projected/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-kube-api-access-2mgzp\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.803878 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.803687    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2d7r\" (UniqueName: \"kubernetes.io/projected/e9ca90f7-2961-4e2d-9345-ec5585c65079-kube-api-access-j2d7r\") pod \"openshift-state-metrics-67dcf8c979-2ws2m\" (UID: \"e9ca90f7-2961-4e2d-9345-ec5585c65079\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"
Sep 15 21:27:10.803878 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.803716    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-root\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.803878 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.803743    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-sys\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.803878 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.803793    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a78378c-9922-4f31-be12-0ad758862a48-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.803878 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.803826    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7a78378c-9922-4f31-be12-0ad758862a48-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.803878 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.803852    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ca90f7-2961-4e2d-9345-ec5585c65079-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-2ws2m\" (UID: \"e9ca90f7-2961-4e2d-9345-ec5585c65079\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"
Sep 15 21:27:10.804107 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.803891    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9ca90f7-2961-4e2d-9345-ec5585c65079-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-2ws2m\" (UID: \"e9ca90f7-2961-4e2d-9345-ec5585c65079\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"
Sep 15 21:27:10.804107 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.803917    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l789\" (UniqueName: \"kubernetes.io/projected/7a78378c-9922-4f31-be12-0ad758862a48-kube-api-access-7l789\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.804107 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.803940    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-metrics-client-ca\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.804107 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.803966    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7a78378c-9922-4f31-be12-0ad758862a48-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.804107 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.803988    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9ca90f7-2961-4e2d-9345-ec5585c65079-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-2ws2m\" (UID: \"e9ca90f7-2961-4e2d-9345-ec5585c65079\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"
Sep 15 21:27:10.804107 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.804007    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.804107 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.804022    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a78378c-9922-4f31-be12-0ad758862a48-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.804107 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.804041    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-node-exporter-textfile\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.804107 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.804055    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-node-exporter-accelerators-collector-config\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.804107 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.804071    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-node-exporter-wtmp\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.804107 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.804088    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a78378c-9922-4f31-be12-0ad758862a48-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.804107 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.804102    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-node-exporter-tls\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.805303 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.805279    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9ca90f7-2961-4e2d-9345-ec5585c65079-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-2ws2m\" (UID: \"e9ca90f7-2961-4e2d-9345-ec5585c65079\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"
Sep 15 21:27:10.807814 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.807791    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e9ca90f7-2961-4e2d-9345-ec5585c65079-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-2ws2m\" (UID: \"e9ca90f7-2961-4e2d-9345-ec5585c65079\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"
Sep 15 21:27:10.811078 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.811055    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e9ca90f7-2961-4e2d-9345-ec5585c65079-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-2ws2m\" (UID: \"e9ca90f7-2961-4e2d-9345-ec5585c65079\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"
Sep 15 21:27:10.814416 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.814375    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2d7r\" (UniqueName: \"kubernetes.io/projected/e9ca90f7-2961-4e2d-9345-ec5585c65079-kube-api-access-j2d7r\") pod \"openshift-state-metrics-67dcf8c979-2ws2m\" (UID: \"e9ca90f7-2961-4e2d-9345-ec5585c65079\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"
Sep 15 21:27:10.905144 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.905104    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l789\" (UniqueName: \"kubernetes.io/projected/7a78378c-9922-4f31-be12-0ad758862a48-kube-api-access-7l789\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.905326 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.905152    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-metrics-client-ca\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.905326 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.905205    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7a78378c-9922-4f31-be12-0ad758862a48-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.905326 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.905231    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.905326 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.905252    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a78378c-9922-4f31-be12-0ad758862a48-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.905326 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.905282    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-node-exporter-textfile\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.905326 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.905306    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-node-exporter-accelerators-collector-config\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.905637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.905334    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-node-exporter-wtmp\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.905637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.905360    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a78378c-9922-4f31-be12-0ad758862a48-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.905637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.905382    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-node-exporter-tls\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.905637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.905414    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mgzp\" (UniqueName: \"kubernetes.io/projected/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-kube-api-access-2mgzp\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.905637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.905443    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-root\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.905637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.905510    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-root\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.905901 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.905729    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-node-exporter-wtmp\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.905978 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.905955    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7a78378c-9922-4f31-be12-0ad758862a48-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.906034 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.905966    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-metrics-client-ca\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.906034 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.906022    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-node-exporter-textfile\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.906190 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.906172    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-sys\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.906245 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.906214    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a78378c-9922-4f31-be12-0ad758862a48-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.906293 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.906250    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7a78378c-9922-4f31-be12-0ad758862a48-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.906293 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.906262    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-node-exporter-accelerators-collector-config\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.906391 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.906343    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-sys\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.906596 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.906570    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7a78378c-9922-4f31-be12-0ad758862a48-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.907191 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.907169    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a78378c-9922-4f31-be12-0ad758862a48-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.908242 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.908212    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.909037 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.909021    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-node-exporter-tls\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.909132 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.909075    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a78378c-9922-4f31-be12-0ad758862a48-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.909132 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.909115    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a78378c-9922-4f31-be12-0ad758862a48-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.915478 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.915439    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l789\" (UniqueName: \"kubernetes.io/projected/7a78378c-9922-4f31-be12-0ad758862a48-kube-api-access-7l789\") pod \"kube-state-metrics-59fbdd77d-sfg9r\" (UID: \"7a78378c-9922-4f31-be12-0ad758862a48\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:10.915848 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.915804    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mgzp\" (UniqueName: \"kubernetes.io/projected/4f0338db-db65-48e6-b35c-e5d7e7bcbb26-kube-api-access-2mgzp\") pod \"node-exporter-q7wqq\" (UID: \"4f0338db-db65-48e6-b35c-e5d7e7bcbb26\") " pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:10.984503 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:10.984410    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"
Sep 15 21:27:11.006244 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.006208    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sqhn2"
Sep 15 21:27:11.017934 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.017908    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"
Sep 15 21:27:11.045892 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.045866    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q7wqq"
Sep 15 21:27:11.052806 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.052774    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b46fs" event={"ID":"117891c4-3453-4091-81f0-46b5532f49b8","Type":"ContainerStarted","Data":"d8a8cb709399fb2c19d77bc67a5683c7a8506fdba2d855ce0e3ae91e3df75cd9"}
Sep 15 21:27:11.053146 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.052829    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-b46fs" event={"ID":"117891c4-3453-4091-81f0-46b5532f49b8","Type":"ContainerStarted","Data":"003468a91e9c16c0b0464e02631a085547094fbbc4661dafce2315c939520217"}
Sep 15 21:27:11.073289 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.072671    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-b46fs" podStartSLOduration=66.907282281 podStartE2EDuration="1m8.072652899s" totalImagesPullingTime="1.165370618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:11.072423984 +0000 UTC m=+68.884432428" watchObservedRunningTime="2026-09-15 21:27:11.072652899 +0000 UTC m=+68.884661340"
Sep 15 21:27:11.279606 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.279536    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m"]
Sep 15 21:27:11.301416 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.301385    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-767bdb48d9-xhf4p"]
Sep 15 21:27:11.304886 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.304860    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.317249 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.317222    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 15 21:27:11.336632 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.336605    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r"]
Sep 15 21:27:11.337479 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.337461    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-767bdb48d9-xhf4p"]
Sep 15 21:27:11.410081 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.409993    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qplsq\" (UniqueName: \"kubernetes.io/projected/ab84fc33-7f81-4825-9766-80fb6487e80e-kube-api-access-qplsq\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.410081 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.410044    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab84fc33-7f81-4825-9766-80fb6487e80e-console-serving-cert\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.410328 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.410122    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-trusted-ca-bundle\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.410328 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.410173    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab84fc33-7f81-4825-9766-80fb6487e80e-console-oauth-config\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.410328 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.410214    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-console-config\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.410328 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.410236    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-oauth-serving-cert\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.410328 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.410270    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-service-ca\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.511407 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.511367    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qplsq\" (UniqueName: \"kubernetes.io/projected/ab84fc33-7f81-4825-9766-80fb6487e80e-kube-api-access-qplsq\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.511559 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.511421    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab84fc33-7f81-4825-9766-80fb6487e80e-console-serving-cert\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.511559 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.511457    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-trusted-ca-bundle\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.511559 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.511484    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab84fc33-7f81-4825-9766-80fb6487e80e-console-oauth-config\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.511559 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.511534    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-console-config\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.511559 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.511554    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-oauth-serving-cert\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.511859 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.511579    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-service-ca\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.512390 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.512358    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-service-ca\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.512520 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.512502    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-console-config\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.512683 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.512666    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-trusted-ca-bundle\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.512949 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.512926    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-oauth-serving-cert\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.515816 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.515792    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab84fc33-7f81-4825-9766-80fb6487e80e-console-oauth-config\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.515910 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.515878    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab84fc33-7f81-4825-9766-80fb6487e80e-console-serving-cert\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.525143 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.525117    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qplsq\" (UniqueName: \"kubernetes.io/projected/ab84fc33-7f81-4825-9766-80fb6487e80e-kube-api-access-qplsq\") pod \"console-767bdb48d9-xhf4p\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") " pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.618039 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.617998    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:11.834805 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.834773    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:27:11.837461 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.837441    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:11.840436 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.840410    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 15 21:27:11.840563 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.840450    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 15 21:27:11.840563 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.840457    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 15 21:27:11.840563 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.840532    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 15 21:27:11.840563 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.840413    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wg5rj\""
Sep 15 21:27:11.840563 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.840423    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 15 21:27:11.840831 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.840411    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 15 21:27:11.840831 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.840822    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 15 21:27:11.840931 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.840860    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 15 21:27:11.840931 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.840905    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 15 21:27:11.862137 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.862052    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:27:11.915010 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.914915    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c19acc79-0bea-422e-8671-f00fd04c7f1e-config-out\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:11.915010 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.914979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:11.915249 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.915015    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c19acc79-0bea-422e-8671-f00fd04c7f1e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:11.915249 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.915076    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbdsk\" (UniqueName: \"kubernetes.io/projected/c19acc79-0bea-422e-8671-f00fd04c7f1e-kube-api-access-xbdsk\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:11.915249 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.915119    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:11.915249 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.915153    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c19acc79-0bea-422e-8671-f00fd04c7f1e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:11.915249 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.915195    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-config-volume\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:11.915249 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.915220    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c19acc79-0bea-422e-8671-f00fd04c7f1e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:11.915249 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.915238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:11.915480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.915291    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c19acc79-0bea-422e-8671-f00fd04c7f1e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:11.915480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.915335    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:11.915480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.915365    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:11.915480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:11.915390    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-web-config\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.016253 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.016204    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.016443 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.016277    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c19acc79-0bea-422e-8671-f00fd04c7f1e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.016443 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.016299    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.016443 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.016322    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.016443 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.016348    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-web-config\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.016443 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.016384    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c19acc79-0bea-422e-8671-f00fd04c7f1e-config-out\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.016443 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.016439    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.016730 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.016470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c19acc79-0bea-422e-8671-f00fd04c7f1e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.016730 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:27:12.016499    2575 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 15 21:27:12.016730 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:27:12.016581    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-main-tls podName:c19acc79-0bea-422e-8671-f00fd04c7f1e nodeName:}" failed. No retries permitted until 2026-09-15 21:27:12.516561972 +0000 UTC m=+70.328570393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c19acc79-0bea-422e-8671-f00fd04c7f1e") : secret "alertmanager-main-tls" not found
Sep 15 21:27:12.016730 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.016503    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbdsk\" (UniqueName: \"kubernetes.io/projected/c19acc79-0bea-422e-8671-f00fd04c7f1e-kube-api-access-xbdsk\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.016730 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.016724    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.016997 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.016748    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c19acc79-0bea-422e-8671-f00fd04c7f1e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.016997 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.016803    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-config-volume\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.016997 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.016830    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c19acc79-0bea-422e-8671-f00fd04c7f1e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.016997 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.016833    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c19acc79-0bea-422e-8671-f00fd04c7f1e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.016997 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:27:12.016995    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c19acc79-0bea-422e-8671-f00fd04c7f1e-alertmanager-trusted-ca-bundle podName:c19acc79-0bea-422e-8671-f00fd04c7f1e nodeName:}" failed. No retries permitted until 2026-09-15 21:27:12.51697786 +0000 UTC m=+70.328986286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c19acc79-0bea-422e-8671-f00fd04c7f1e-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "c19acc79-0bea-422e-8671-f00fd04c7f1e") : configmap references non-existent config key: ca-bundle.crt
Sep 15 21:27:12.018552 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.018232    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c19acc79-0bea-422e-8671-f00fd04c7f1e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.019904 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.019857    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.019993 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.019954    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c19acc79-0bea-422e-8671-f00fd04c7f1e-config-out\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.020062 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.020037    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c19acc79-0bea-422e-8671-f00fd04c7f1e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.020136 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.020038    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-config-volume\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.020773 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.020729    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.021185 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.021165    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.021263 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.021205    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.021448 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.021427    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-web-config\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.025461 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.025436    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbdsk\" (UniqueName: \"kubernetes.io/projected/c19acc79-0bea-422e-8671-f00fd04c7f1e-kube-api-access-xbdsk\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.059015 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.058909    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7wqq" event={"ID":"4f0338db-db65-48e6-b35c-e5d7e7bcbb26","Type":"ContainerStarted","Data":"18aa082e646d2415f07cf061d7863a92ed798aa7286a4d97b4dc98c8a8cf23c3"}
Sep 15 21:27:12.278072 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:27:12.277995    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a78378c_9922_4f31_be12_0ad758862a48.slice/crio-c3c4ab208fd643a5edda97b379842d22e4565884e8e2031e4c0d3049338d030a WatchSource:0}: Error finding container c3c4ab208fd643a5edda97b379842d22e4565884e8e2031e4c0d3049338d030a: Status 404 returned error can't find the container with id c3c4ab208fd643a5edda97b379842d22e4565884e8e2031e4c0d3049338d030a
Sep 15 21:27:12.278876 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:27:12.278851    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ca90f7_2961_4e2d_9345_ec5585c65079.slice/crio-e6befa550d3fbf66b1e12bdd14f19610454e2fd1d277f5441a34e00ec795ca58 WatchSource:0}: Error finding container e6befa550d3fbf66b1e12bdd14f19610454e2fd1d277f5441a34e00ec795ca58: Status 404 returned error can't find the container with id e6befa550d3fbf66b1e12bdd14f19610454e2fd1d277f5441a34e00ec795ca58
Sep 15 21:27:12.522137 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.522081    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c19acc79-0bea-422e-8671-f00fd04c7f1e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.522341 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.522179    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.523361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.523335    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c19acc79-0bea-422e-8671-f00fd04c7f1e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.525766 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.525723    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:12.751421 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:12.751389    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:27:13.064230 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:13.064143    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r" event={"ID":"7a78378c-9922-4f31-be12-0ad758862a48","Type":"ContainerStarted","Data":"c3c4ab208fd643a5edda97b379842d22e4565884e8e2031e4c0d3049338d030a"}
Sep 15 21:27:13.065538 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:13.065512    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m" event={"ID":"e9ca90f7-2961-4e2d-9345-ec5585c65079","Type":"ContainerStarted","Data":"e6befa550d3fbf66b1e12bdd14f19610454e2fd1d277f5441a34e00ec795ca58"}
Sep 15 21:27:13.835642 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:13.835015    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:13.835642 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:13.835362    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:13.841181 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:13.841140    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:13.955075 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:13.954969    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-767bdb48d9-xhf4p"]
Sep 15 21:27:13.963820 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:27:13.963787    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab84fc33_7f81_4825_9766_80fb6487e80e.slice/crio-895173309db265698bb2c1fe9521359eb397f7705f05d6692ef371bbfc5d25da WatchSource:0}: Error finding container 895173309db265698bb2c1fe9521359eb397f7705f05d6692ef371bbfc5d25da: Status 404 returned error can't find the container with id 895173309db265698bb2c1fe9521359eb397f7705f05d6692ef371bbfc5d25da
Sep 15 21:27:14.073814 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.073653    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-767bdb48d9-xhf4p" event={"ID":"ab84fc33-7f81-4825-9766-80fb6487e80e","Type":"ContainerStarted","Data":"cc16be1fa494e5bcd5218e0d022f5e065170213d9487b8378c30a62c55d745f1"}
Sep 15 21:27:14.073814 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.073712    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-767bdb48d9-xhf4p" event={"ID":"ab84fc33-7f81-4825-9766-80fb6487e80e","Type":"ContainerStarted","Data":"895173309db265698bb2c1fe9521359eb397f7705f05d6692ef371bbfc5d25da"}
Sep 15 21:27:14.077440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.077311    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rsbdn" event={"ID":"f508cca8-4d27-4ee5-926d-1a7a3ce42286","Type":"ContainerStarted","Data":"ceba71bea7423907630a5b0f189462eb5c25f0954ecbc077ed749f5cae7340fa"}
Sep 15 21:27:14.077440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.077373    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:27:14.082440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.081942    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7wqq" event={"ID":"4f0338db-db65-48e6-b35c-e5d7e7bcbb26","Type":"ContainerStarted","Data":"f0160a62f95bc04b67724f95c4260962e66ac25ee81c4ac4a22ef4eda54f9af0"}
Sep 15 21:27:14.086232 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.085743    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m" event={"ID":"e9ca90f7-2961-4e2d-9345-ec5585c65079","Type":"ContainerStarted","Data":"df619f98c32d2348ab073d62f828c46fe227d76a19bf50435ceab534bfa269a1"}
Sep 15 21:27:14.092329 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.092271    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:14.097555 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.097481    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rsbdn" podStartSLOduration=66.111381788 podStartE2EDuration="1m11.097455815s" totalImagesPullingTime="4.986074027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:14.096863837 +0000 UTC m=+71.908872279" watchObservedRunningTime="2026-09-15 21:27:14.097455815 +0000 UTC m=+71.909464279"
Sep 15 21:27:14.212729 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.212624    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:27:14.240634 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:27:14.240597    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19acc79_0bea_422e_8671_f00fd04c7f1e.slice/crio-707521a4900559334fe25cca648ea699928124ea4bd78c87b13db21c49b68877 WatchSource:0}: Error finding container 707521a4900559334fe25cca648ea699928124ea4bd78c87b13db21c49b68877: Status 404 returned error can't find the container with id 707521a4900559334fe25cca648ea699928124ea4bd78c87b13db21c49b68877
Sep 15 21:27:14.786428 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.786401    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-865dc86548-frw4b"]
Sep 15 21:27:14.801178 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.801151    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-865dc86548-frw4b"]
Sep 15 21:27:14.801314 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.801286    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.803504 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.803481    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 15 21:27:14.803649 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.803535    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 15 21:27:14.803649 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.803603    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dvpig79p45k45\""
Sep 15 21:27:14.803649 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.803613    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 15 21:27:14.803649 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.803636    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-ddz6k\""
Sep 15 21:27:14.803855 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.803656    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 15 21:27:14.803855 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.803738    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 15 21:27:14.845935 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.845910    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.846066 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.845942    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-grpc-tls\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.846066 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.845970    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxztx\" (UniqueName: \"kubernetes.io/projected/6baef592-84e4-408d-8ee2-8638d6003d7f-kube-api-access-jxztx\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.846066 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.846016    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.846066 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.846055    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.846203 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.846072    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.846203 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.846113    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-thanos-querier-tls\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.846203 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.846143    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6baef592-84e4-408d-8ee2-8638d6003d7f-metrics-client-ca\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.947318 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.947289    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.947460 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.947329    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.947460 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.947379    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-thanos-querier-tls\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.947460 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.947423    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6baef592-84e4-408d-8ee2-8638d6003d7f-metrics-client-ca\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.947612 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.947541    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.947659 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.947573    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-grpc-tls\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.947706 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.947672    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxztx\" (UniqueName: \"kubernetes.io/projected/6baef592-84e4-408d-8ee2-8638d6003d7f-kube-api-access-jxztx\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.947784 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.947702    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.948387 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.948337    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6baef592-84e4-408d-8ee2-8638d6003d7f-metrics-client-ca\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.950005 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.949981    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.950161 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.950132    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.950563 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.950549    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-grpc-tls\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.950614 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.950566    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-thanos-querier-tls\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.950614 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.950594    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.950681 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.950623    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6baef592-84e4-408d-8ee2-8638d6003d7f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:14.956810 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:14.956787    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxztx\" (UniqueName: \"kubernetes.io/projected/6baef592-84e4-408d-8ee2-8638d6003d7f-kube-api-access-jxztx\") pod \"thanos-querier-865dc86548-frw4b\" (UID: \"6baef592-84e4-408d-8ee2-8638d6003d7f\") " pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:15.090710 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.090620    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c19acc79-0bea-422e-8671-f00fd04c7f1e","Type":"ContainerStarted","Data":"707521a4900559334fe25cca648ea699928124ea4bd78c87b13db21c49b68877"}
Sep 15 21:27:15.092116 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.092087    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v6j25" event={"ID":"21784d6c-c661-42eb-8b09-41376dacc1a3","Type":"ContainerStarted","Data":"25503557d7015fdb6f5c3843ffbb06a6b9f95c10c6290c10e55c78507f10d94b"}
Sep 15 21:27:15.093615 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.093586    2575 generic.go:356] "Generic (PLEG): container finished" podID="4f0338db-db65-48e6-b35c-e5d7e7bcbb26" containerID="f0160a62f95bc04b67724f95c4260962e66ac25ee81c4ac4a22ef4eda54f9af0" exitCode=0
Sep 15 21:27:15.093736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.093660    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7wqq" event={"ID":"4f0338db-db65-48e6-b35c-e5d7e7bcbb26","Type":"ContainerDied","Data":"f0160a62f95bc04b67724f95c4260962e66ac25ee81c4ac4a22ef4eda54f9af0"}
Sep 15 21:27:15.099282 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.099256    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m" event={"ID":"e9ca90f7-2961-4e2d-9345-ec5585c65079","Type":"ContainerStarted","Data":"9fe4e67372f3d39c66ee9cf035278c7c95efac2a1fab68bb0f57f049b30e2fa5"}
Sep 15 21:27:15.110062 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.109927    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v6j25" podStartSLOduration=67.077204514 podStartE2EDuration="1m12.109910835s" totalImagesPullingTime="5.032706321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:15.109520798 +0000 UTC m=+72.921529240" watchObservedRunningTime="2026-09-15 21:27:15.109910835 +0000 UTC m=+72.921919257"
Sep 15 21:27:15.111004 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.110982    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:15.143548 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.143504    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-767bdb48d9-xhf4p" podStartSLOduration=4.14348967 podStartE2EDuration="4.14348967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:15.142653639 +0000 UTC m=+72.954662097" watchObservedRunningTime="2026-09-15 21:27:15.14348967 +0000 UTC m=+72.955498109"
Sep 15 21:27:15.183868 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.183834    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6fb596986f-np7bm"]
Sep 15 21:27:15.202184 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.202153    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6fb596986f-np7bm"]
Sep 15 21:27:15.202331 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.202260    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.204780 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.204729    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 15 21:27:15.205145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.204729    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 15 21:27:15.205145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.204929    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 15 21:27:15.205145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.204997    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9ufsd368muqpg\""
Sep 15 21:27:15.205145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.205130    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bzwhp\""
Sep 15 21:27:15.205373 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.205359    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 15 21:27:15.250705 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.250637    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-metrics-server-audit-profiles\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.250705 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.250687    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-client-ca-bundle\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.251055 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.250821    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-audit-log\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.251055 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.250864    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-secret-metrics-server-client-certs\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.251055 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.250937    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-secret-metrics-server-tls\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.251055 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.251012    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.251342 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.251082    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-542hs\" (UniqueName: \"kubernetes.io/projected/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-kube-api-access-542hs\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.352476 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.352441    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-audit-log\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.352692 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.352494    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-secret-metrics-server-client-certs\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.352692 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.352566    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-secret-metrics-server-tls\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.352692 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.352592    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.352692 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.352614    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-542hs\" (UniqueName: \"kubernetes.io/projected/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-kube-api-access-542hs\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.352692 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.352653    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-metrics-server-audit-profiles\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.352692 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.352686    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-client-ca-bundle\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.354867 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.353230    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-audit-log\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.354867 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.354163    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.354867 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.354673    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-metrics-server-audit-profiles\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.360143 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.359644    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-client-ca-bundle\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.360143 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.360096    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-secret-metrics-server-tls\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.362892 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.362864    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-secret-metrics-server-client-certs\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.364253 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.364209    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-542hs\" (UniqueName: \"kubernetes.io/projected/e0e2f419-a8ae-466a-8d60-9c2c7805c1b5-kube-api-access-542hs\") pod \"metrics-server-6fb596986f-np7bm\" (UID: \"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5\") " pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.455516 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.454680    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-zs2l2"]
Sep 15 21:27:15.484002 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.483189    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-zs2l2"]
Sep 15 21:27:15.484002 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.483236    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-865dc86548-frw4b"]
Sep 15 21:27:15.484002 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.483378    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zs2l2"
Sep 15 21:27:15.486603 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.486041    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-f2584\""
Sep 15 21:27:15.486603 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.486318    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 15 21:27:15.489489 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:27:15.489466    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6baef592_84e4_408d_8ee2_8638d6003d7f.slice/crio-9b2bc92ecf7a6c430f103fe98c4d8b5c4c2e7a77cc0418f5f436c97de8ed9640 WatchSource:0}: Error finding container 9b2bc92ecf7a6c430f103fe98c4d8b5c4c2e7a77cc0418f5f436c97de8ed9640: Status 404 returned error can't find the container with id 9b2bc92ecf7a6c430f103fe98c4d8b5c4c2e7a77cc0418f5f436c97de8ed9640
Sep 15 21:27:15.520941 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.520788    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:15.554693 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.554662    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv7pm\" (UniqueName: \"kubernetes.io/projected/563da805-dd74-457e-81e4-b82796a10106-kube-api-access-cv7pm\") pod \"monitoring-plugin-5d85778b9d-zs2l2\" (UID: \"563da805-dd74-457e-81e4-b82796a10106\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zs2l2"
Sep 15 21:27:15.554823 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.554726    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/563da805-dd74-457e-81e4-b82796a10106-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-zs2l2\" (UID: \"563da805-dd74-457e-81e4-b82796a10106\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zs2l2"
Sep 15 21:27:15.655775 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.655678    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv7pm\" (UniqueName: \"kubernetes.io/projected/563da805-dd74-457e-81e4-b82796a10106-kube-api-access-cv7pm\") pod \"monitoring-plugin-5d85778b9d-zs2l2\" (UID: \"563da805-dd74-457e-81e4-b82796a10106\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zs2l2"
Sep 15 21:27:15.655775 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.655744    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/563da805-dd74-457e-81e4-b82796a10106-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-zs2l2\" (UID: \"563da805-dd74-457e-81e4-b82796a10106\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zs2l2"
Sep 15 21:27:15.658939 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.658907    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/563da805-dd74-457e-81e4-b82796a10106-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-zs2l2\" (UID: \"563da805-dd74-457e-81e4-b82796a10106\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zs2l2"
Sep 15 21:27:15.677073 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.677041    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv7pm\" (UniqueName: \"kubernetes.io/projected/563da805-dd74-457e-81e4-b82796a10106-kube-api-access-cv7pm\") pod \"monitoring-plugin-5d85778b9d-zs2l2\" (UID: \"563da805-dd74-457e-81e4-b82796a10106\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zs2l2"
Sep 15 21:27:15.798243 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.798208    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zs2l2"
Sep 15 21:27:15.826845 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:15.826819    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6fb596986f-np7bm"]
Sep 15 21:27:15.848731 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:27:15.848701    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0e2f419_a8ae_466a_8d60_9c2c7805c1b5.slice/crio-33838e4c1f114f0d4ca59c4cc672e8b93219ed7dffcb92c4812bb19539da25ee WatchSource:0}: Error finding container 33838e4c1f114f0d4ca59c4cc672e8b93219ed7dffcb92c4812bb19539da25ee: Status 404 returned error can't find the container with id 33838e4c1f114f0d4ca59c4cc672e8b93219ed7dffcb92c4812bb19539da25ee
Sep 15 21:27:16.105508 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.105437    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7wqq" event={"ID":"4f0338db-db65-48e6-b35c-e5d7e7bcbb26","Type":"ContainerStarted","Data":"d4ac526db3a6aaa58da60935652fcd68dc2326689dec28748309361756db8537"}
Sep 15 21:27:16.105508 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.105493    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q7wqq" event={"ID":"4f0338db-db65-48e6-b35c-e5d7e7bcbb26","Type":"ContainerStarted","Data":"aba665aefa2924d687fe4cd21a019365f021ccfd994f059f88307417f02d7ecc"}
Sep 15 21:27:16.106687 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.106665    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6fb596986f-np7bm" event={"ID":"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5","Type":"ContainerStarted","Data":"33838e4c1f114f0d4ca59c4cc672e8b93219ed7dffcb92c4812bb19539da25ee"}
Sep 15 21:27:16.107919 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.107893    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-865dc86548-frw4b" event={"ID":"6baef592-84e4-408d-8ee2-8638d6003d7f","Type":"ContainerStarted","Data":"9b2bc92ecf7a6c430f103fe98c4d8b5c4c2e7a77cc0418f5f436c97de8ed9640"}
Sep 15 21:27:16.110010 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.109985    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r" event={"ID":"7a78378c-9922-4f31-be12-0ad758862a48","Type":"ContainerStarted","Data":"30badd0b3b553ebb4fed2f11740537487f9e5735f395058f9897ef6a9ef7f84a"}
Sep 15 21:27:16.110093 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.110042    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r" event={"ID":"7a78378c-9922-4f31-be12-0ad758862a48","Type":"ContainerStarted","Data":"cce90510f45c7914602fc433936a336fca7f59789dd5ec0e0ab8adb4c9dd8646"}
Sep 15 21:27:16.110093 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.110064    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r" event={"ID":"7a78378c-9922-4f31-be12-0ad758862a48","Type":"ContainerStarted","Data":"1ab6c5e0ff0630bc7ca8bc623b4517814f52a2b291a00469cccb1330caa131c8"}
Sep 15 21:27:16.123803 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.123730    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q7wqq" podStartSLOduration=3.116490802 podStartE2EDuration="6.123709495s" totalImagesPullingTime="2.888577684s" totalInitContainerRuntime="118.641009ms" isStatefulPod=true podCreationTimestamp="2026-09-15 21:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:16.122957116 +0000 UTC m=+73.934965560" watchObservedRunningTime="2026-09-15 21:27:16.123709495 +0000 UTC m=+73.935717939"
Sep 15 21:27:16.142357 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.142310    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-sfg9r" podStartSLOduration=3.256530247 podStartE2EDuration="6.142289021s" totalImagesPullingTime="2.885758774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:16.141067966 +0000 UTC m=+73.953076411" watchObservedRunningTime="2026-09-15 21:27:16.142289021 +0000 UTC m=+73.954297460"
Sep 15 21:27:16.390985 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.390890    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-zs2l2"]
Sep 15 21:27:16.394554 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:27:16.394522    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563da805_dd74_457e_81e4_b82796a10106.slice/crio-a2ae930ed4caa14e4f13f6acc89e26e1a16aa6bc034fed24a2b8ce6043dff514 WatchSource:0}: Error finding container a2ae930ed4caa14e4f13f6acc89e26e1a16aa6bc034fed24a2b8ce6043dff514: Status 404 returned error can't find the container with id a2ae930ed4caa14e4f13f6acc89e26e1a16aa6bc034fed24a2b8ce6043dff514
Sep 15 21:27:16.969148 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.969111    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:27:16.974749 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.974723    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:16.977233 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.977206    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 21:27:16.979017 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.978264    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 21:27:16.979017 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.978530    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 21:27:16.979017 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.978790    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 21:27:16.979017 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.978852    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dcjrj\""
Sep 15 21:27:16.979017 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.978884    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 21:27:16.979017 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.978970    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 21:27:16.979397 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.979191    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 21:27:16.979397 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.979354    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 21:27:16.981274 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.979535    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 21:27:16.981274 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.979664    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 21:27:16.981274 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.979718    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 21:27:16.981274 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.979840    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 21:27:16.981274 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.979864    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 21:27:16.981274 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.980091    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9jlkd9tn1go9h\""
Sep 15 21:27:16.982044 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.981880    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 21:27:16.991659 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:16.991635    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:27:17.070957 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.070913    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df0511db-812d-4853-b5d3-810ad2b50fb2-config-out\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.070968    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.070994    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071029    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071053    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071077    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071132    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df0511db-812d-4853-b5d3-810ad2b50fb2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071155    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071190    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071214    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071244    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-config\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071267    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071288    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-web-config\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071317    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071349    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwgzl\" (UniqueName: \"kubernetes.io/projected/df0511db-812d-4853-b5d3-810ad2b50fb2-kube-api-access-pwgzl\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071392    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071420    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.071486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071444    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.072018 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.071492    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.106929 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.106888    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-767bdb48d9-xhf4p"]
Sep 15 21:27:17.117921 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.117857    2575 generic.go:356] "Generic (PLEG): container finished" podID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerID="4f3cc0b7caeb1229d1e9995418d17053dce3e7e73cb0cff12fba1a3fe6dfb30c" exitCode=0
Sep 15 21:27:17.118256 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.117969    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c19acc79-0bea-422e-8671-f00fd04c7f1e","Type":"ContainerDied","Data":"4f3cc0b7caeb1229d1e9995418d17053dce3e7e73cb0cff12fba1a3fe6dfb30c"}
Sep 15 21:27:17.119931 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.119828    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zs2l2" event={"ID":"563da805-dd74-457e-81e4-b82796a10106","Type":"ContainerStarted","Data":"a2ae930ed4caa14e4f13f6acc89e26e1a16aa6bc034fed24a2b8ce6043dff514"}
Sep 15 21:27:17.123692 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.123669    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m" event={"ID":"e9ca90f7-2961-4e2d-9345-ec5585c65079","Type":"ContainerStarted","Data":"980a244011d93f89e158f789496973483e80cfda8a6281c6f68e1999980a1dfb"}
Sep 15 21:27:17.172112 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172072    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172112 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172111    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172112 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172151    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-config\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172413 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172169    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172413 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172184    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-web-config\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172413 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172219    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172413 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwgzl\" (UniqueName: \"kubernetes.io/projected/df0511db-812d-4853-b5d3-810ad2b50fb2-kube-api-access-pwgzl\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172413 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172341    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172413 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172386    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172413 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172403    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172739 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172739 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172557    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df0511db-812d-4853-b5d3-810ad2b50fb2-config-out\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172739 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172628    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172739 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172643    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172739 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172669    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172739 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172699    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172739 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172717    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172739 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172784    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.172739 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172800    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df0511db-812d-4853-b5d3-810ad2b50fb2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.173281 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.172822    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.173792 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.173741    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.174018 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.174000    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.175468 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.175445    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.178546 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.175957    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.178546 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.176182    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.184721 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.180491    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.184721 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.182563    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-config\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.184721 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.183332    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.184721 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.184092    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.187859 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.187833    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.194058 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.194033    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.195542 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.195478    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.197485 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.197439    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwgzl\" (UniqueName: \"kubernetes.io/projected/df0511db-812d-4853-b5d3-810ad2b50fb2-kube-api-access-pwgzl\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.198618 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.198600    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df0511db-812d-4853-b5d3-810ad2b50fb2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.201216 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.201192    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.204016 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.203973    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.205227 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.204926    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.205227 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.204970    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-web-config\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.205227 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.205109    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.205227 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.205180    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df0511db-812d-4853-b5d3-810ad2b50fb2-config-out\") pod \"prometheus-k8s-0\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:17.220673 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.220566    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-2ws2m" podStartSLOduration=5.20858237 podStartE2EDuration="7.220546679s" totalImagesPullingTime="2.011964309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:17.219205891 +0000 UTC m=+75.031214334" watchObservedRunningTime="2026-09-15 21:27:17.220546679 +0000 UTC m=+75.032555125"
Sep 15 21:27:17.293662 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:17.293328    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:18.923076 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:18.923049    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:27:19.134235 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:19.134143    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zs2l2" event={"ID":"563da805-dd74-457e-81e4-b82796a10106","Type":"ContainerStarted","Data":"f4195efe7d911dc79c889315db5b0b9e5f00fe7f1bad4bec38b234c2330e18e4"}
Sep 15 21:27:19.134394 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:19.134379    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zs2l2"
Sep 15 21:27:19.135998 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:19.135971    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6fb596986f-np7bm" event={"ID":"e0e2f419-a8ae-466a-8d60-9c2c7805c1b5","Type":"ContainerStarted","Data":"732ec681d950d767606932ff2b38fa269281ec505b2644496593831d5a598283"}
Sep 15 21:27:19.138986 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:19.138961    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-865dc86548-frw4b" event={"ID":"6baef592-84e4-408d-8ee2-8638d6003d7f","Type":"ContainerStarted","Data":"92a7a91622e4473dde529dbff458d686edb49dd7f04f03377332d8647706384a"}
Sep 15 21:27:19.139101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:19.139034    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-865dc86548-frw4b" event={"ID":"6baef592-84e4-408d-8ee2-8638d6003d7f","Type":"ContainerStarted","Data":"e4f92f40e60644fe14df5369c5901a81637dcf552d06999307601a944581c26c"}
Sep 15 21:27:19.139101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:19.139071    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-865dc86548-frw4b" event={"ID":"6baef592-84e4-408d-8ee2-8638d6003d7f","Type":"ContainerStarted","Data":"4eeae8b786039834c373ea34829bc9b1eb041db80ba3782b77e1c8b36eafc051"}
Sep 15 21:27:19.140694 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:19.140481    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zs2l2"
Sep 15 21:27:19.168671 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:19.168627    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6fb596986f-np7bm" podStartSLOduration=1.378471741 podStartE2EDuration="4.168614089s" totalImagesPullingTime="2.790142348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:19.168359411 +0000 UTC m=+76.980367853" watchObservedRunningTime="2026-09-15 21:27:19.168614089 +0000 UTC m=+76.980622524"
Sep 15 21:27:19.169363 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:19.169338    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-zs2l2" podStartSLOduration=1.924371337 podStartE2EDuration="4.169331337s" totalImagesPullingTime="2.24496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:19.150133311 +0000 UTC m=+76.962141757" watchObservedRunningTime="2026-09-15 21:27:19.169331337 +0000 UTC m=+76.981339782"
Sep 15 21:27:19.244080 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:27:19.244049    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0511db_812d_4853_b5d3_810ad2b50fb2.slice/crio-fa5bcb312f6ed0340b02992e15f653fe8efdc987ad5091a615f88b98dd001700 WatchSource:0}: Error finding container fa5bcb312f6ed0340b02992e15f653fe8efdc987ad5091a615f88b98dd001700: Status 404 returned error can't find the container with id fa5bcb312f6ed0340b02992e15f653fe8efdc987ad5091a615f88b98dd001700
Sep 15 21:27:20.147010 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:20.146961    2575 generic.go:356] "Generic (PLEG): container finished" podID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerID="5b78d669ad6a69ddf69f2fed14b35228b8bd815f4900c812aaafcec4507cb8fa" exitCode=0
Sep 15 21:27:20.147462 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:20.147125    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df0511db-812d-4853-b5d3-810ad2b50fb2","Type":"ContainerDied","Data":"5b78d669ad6a69ddf69f2fed14b35228b8bd815f4900c812aaafcec4507cb8fa"}
Sep 15 21:27:20.147462 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:20.147238    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df0511db-812d-4853-b5d3-810ad2b50fb2","Type":"ContainerStarted","Data":"fa5bcb312f6ed0340b02992e15f653fe8efdc987ad5091a615f88b98dd001700"}
Sep 15 21:27:20.151497 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:20.151473    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c19acc79-0bea-422e-8671-f00fd04c7f1e","Type":"ContainerStarted","Data":"636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310"}
Sep 15 21:27:20.151589 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:20.151545    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c19acc79-0bea-422e-8671-f00fd04c7f1e","Type":"ContainerStarted","Data":"d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437"}
Sep 15 21:27:20.151733 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:20.151588    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c19acc79-0bea-422e-8671-f00fd04c7f1e","Type":"ContainerStarted","Data":"68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5"}
Sep 15 21:27:20.151733 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:20.151625    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c19acc79-0bea-422e-8671-f00fd04c7f1e","Type":"ContainerStarted","Data":"7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec"}
Sep 15 21:27:20.151733 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:20.151666    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c19acc79-0bea-422e-8671-f00fd04c7f1e","Type":"ContainerStarted","Data":"ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2"}
Sep 15 21:27:21.161733 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:21.161694    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c19acc79-0bea-422e-8671-f00fd04c7f1e","Type":"ContainerStarted","Data":"05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5"}
Sep 15 21:27:21.164819 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:21.164692    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-865dc86548-frw4b" event={"ID":"6baef592-84e4-408d-8ee2-8638d6003d7f","Type":"ContainerStarted","Data":"e317a4de7f4bacfa308818fa430c25f752b3a51d78cadb6d2e6a744c033b3ae6"}
Sep 15 21:27:21.164931 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:21.164849    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-865dc86548-frw4b" event={"ID":"6baef592-84e4-408d-8ee2-8638d6003d7f","Type":"ContainerStarted","Data":"e4ba5d4fb62dc7d540b1b05cc26239150728ea260293b9ef49ed15940e550dc1"}
Sep 15 21:27:21.164931 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:21.164884    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-865dc86548-frw4b" event={"ID":"6baef592-84e4-408d-8ee2-8638d6003d7f","Type":"ContainerStarted","Data":"1adaf60b30c2021d92033950c1e6761766e9cf6b65cbc06bc588d7d137449de1"}
Sep 15 21:27:21.187330 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:21.187282    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.341064904 podStartE2EDuration="10.187265067s" totalImagesPullingTime="4.8307874s" totalInitContainerRuntime="15.412763ms" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:21.185603937 +0000 UTC m=+78.997612393" watchObservedRunningTime="2026-09-15 21:27:21.187265067 +0000 UTC m=+78.999273511"
Sep 15 21:27:21.210337 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:21.210182    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-865dc86548-frw4b" podStartSLOduration=2.5680341110000002 podStartE2EDuration="7.210159474s" totalImagesPullingTime="4.642125363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:21.209056165 +0000 UTC m=+79.021064609" watchObservedRunningTime="2026-09-15 21:27:21.210159474 +0000 UTC m=+79.022167928"
Sep 15 21:27:21.618876 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:21.618837    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:22.170527 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:22.170493    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:23.181923 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:23.181892    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-865dc86548-frw4b"
Sep 15 21:27:24.183034 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:24.182988    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df0511db-812d-4853-b5d3-810ad2b50fb2","Type":"ContainerStarted","Data":"0d6bd888acb6729a983c9b3786cbab68f6e1b39c4daa39d179ed8f27fc6ee337"}
Sep 15 21:27:24.183475 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:24.183105    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df0511db-812d-4853-b5d3-810ad2b50fb2","Type":"ContainerStarted","Data":"cdad8bb0cb9786f1d1d824816b5c6661dd44477ff7de9dbae82a787a961e73f8"}
Sep 15 21:27:24.183475 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:24.183154    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df0511db-812d-4853-b5d3-810ad2b50fb2","Type":"ContainerStarted","Data":"42cdd07feb6f6d7536e16e3649b7c249fedf721391aec10ae845b834f143cee0"}
Sep 15 21:27:24.183475 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:24.183194    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df0511db-812d-4853-b5d3-810ad2b50fb2","Type":"ContainerStarted","Data":"ebbfbaf07d92fc33075d97e2a55c3faf5a728058ad32f8a60b6f6fe5d3295277"}
Sep 15 21:27:24.183475 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:24.183251    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df0511db-812d-4853-b5d3-810ad2b50fb2","Type":"ContainerStarted","Data":"b75426f6767bc0cf82e7eb54852f7adcbd5bcbc89a9cb6b0ee017facaa22145e"}
Sep 15 21:27:24.183475 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:24.183287    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df0511db-812d-4853-b5d3-810ad2b50fb2","Type":"ContainerStarted","Data":"0611236b679062a770c8e119b041fe53b5bb5df68e8cf2cc5ac8c3d9eee71647"}
Sep 15 21:27:24.210476 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:24.210419    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.79970006 podStartE2EDuration="8.210397712s" totalImagesPullingTime="3.389289158s" totalInitContainerRuntime="21.408494ms" isStatefulPod=false podCreationTimestamp="2026-09-15 21:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:27:24.21000376 +0000 UTC m=+82.022012205" watchObservedRunningTime="2026-09-15 21:27:24.210397712 +0000 UTC m=+82.022406156"
Sep 15 21:27:27.293836 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:27.293806    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:27:28.523430 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:28.523394    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8dcdf8955-425x9"]
Sep 15 21:27:35.521707 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:35.521652    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:35.522120 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:35.521738    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:42.148224 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.148164    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-767bdb48d9-xhf4p" podUID="ab84fc33-7f81-4825-9766-80fb6487e80e" containerName="console" containerID="cri-o://cc16be1fa494e5bcd5218e0d022f5e065170213d9487b8378c30a62c55d745f1" gracePeriod=15
Sep 15 21:27:42.263017 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.262989    2575 generic.go:356] "Generic (PLEG): container finished" podID="ab84fc33-7f81-4825-9766-80fb6487e80e" containerID="cc16be1fa494e5bcd5218e0d022f5e065170213d9487b8378c30a62c55d745f1" exitCode=0
Sep 15 21:27:42.263106 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.263061    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-767bdb48d9-xhf4p" event={"ID":"ab84fc33-7f81-4825-9766-80fb6487e80e","Type":"ContainerDied","Data":"cc16be1fa494e5bcd5218e0d022f5e065170213d9487b8378c30a62c55d745f1"}
Sep 15 21:27:42.374109 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.374089    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:42.504039 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.503995    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab84fc33-7f81-4825-9766-80fb6487e80e-console-oauth-config\") pod \"ab84fc33-7f81-4825-9766-80fb6487e80e\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") "
Sep 15 21:27:42.504211 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.504059    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-console-config\") pod \"ab84fc33-7f81-4825-9766-80fb6487e80e\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") "
Sep 15 21:27:42.504211 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.504085    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qplsq\" (UniqueName: \"kubernetes.io/projected/ab84fc33-7f81-4825-9766-80fb6487e80e-kube-api-access-qplsq\") pod \"ab84fc33-7f81-4825-9766-80fb6487e80e\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") "
Sep 15 21:27:42.504211 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.504125    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab84fc33-7f81-4825-9766-80fb6487e80e-console-serving-cert\") pod \"ab84fc33-7f81-4825-9766-80fb6487e80e\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") "
Sep 15 21:27:42.504211 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.504153    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-service-ca\") pod \"ab84fc33-7f81-4825-9766-80fb6487e80e\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") "
Sep 15 21:27:42.504211 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.504181    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-trusted-ca-bundle\") pod \"ab84fc33-7f81-4825-9766-80fb6487e80e\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") "
Sep 15 21:27:42.504443 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.504216    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-oauth-serving-cert\") pod \"ab84fc33-7f81-4825-9766-80fb6487e80e\" (UID: \"ab84fc33-7f81-4825-9766-80fb6487e80e\") "
Sep 15 21:27:42.504534 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.504511    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-console-config" (OuterVolumeSpecName: "console-config") pod "ab84fc33-7f81-4825-9766-80fb6487e80e" (UID: "ab84fc33-7f81-4825-9766-80fb6487e80e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:27:42.504593 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.504530    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-service-ca" (OuterVolumeSpecName: "service-ca") pod "ab84fc33-7f81-4825-9766-80fb6487e80e" (UID: "ab84fc33-7f81-4825-9766-80fb6487e80e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:27:42.504694 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.504671    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "ab84fc33-7f81-4825-9766-80fb6487e80e" (UID: "ab84fc33-7f81-4825-9766-80fb6487e80e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:27:42.504743 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.504702    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "ab84fc33-7f81-4825-9766-80fb6487e80e" (UID: "ab84fc33-7f81-4825-9766-80fb6487e80e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:27:42.506434 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.506412    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab84fc33-7f81-4825-9766-80fb6487e80e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "ab84fc33-7f81-4825-9766-80fb6487e80e" (UID: "ab84fc33-7f81-4825-9766-80fb6487e80e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:27:42.506850 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.506827    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab84fc33-7f81-4825-9766-80fb6487e80e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "ab84fc33-7f81-4825-9766-80fb6487e80e" (UID: "ab84fc33-7f81-4825-9766-80fb6487e80e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:27:42.506850 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.506834    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab84fc33-7f81-4825-9766-80fb6487e80e-kube-api-access-qplsq" (OuterVolumeSpecName: "kube-api-access-qplsq") pod "ab84fc33-7f81-4825-9766-80fb6487e80e" (UID: "ab84fc33-7f81-4825-9766-80fb6487e80e"). InnerVolumeSpecName "kube-api-access-qplsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:27:42.605339 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.605308    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ab84fc33-7f81-4825-9766-80fb6487e80e-console-oauth-config\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:27:42.605339 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.605336    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-console-config\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:27:42.605339 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.605345    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qplsq\" (UniqueName: \"kubernetes.io/projected/ab84fc33-7f81-4825-9766-80fb6487e80e-kube-api-access-qplsq\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:27:42.605545 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.605354    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ab84fc33-7f81-4825-9766-80fb6487e80e-console-serving-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:27:42.605545 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.605362    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:27:42.605545 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.605370    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-trusted-ca-bundle\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:27:42.605545 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:42.605377    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ab84fc33-7f81-4825-9766-80fb6487e80e-oauth-serving-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:27:43.268413 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:43.268376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-767bdb48d9-xhf4p" event={"ID":"ab84fc33-7f81-4825-9766-80fb6487e80e","Type":"ContainerDied","Data":"895173309db265698bb2c1fe9521359eb397f7705f05d6692ef371bbfc5d25da"}
Sep 15 21:27:43.268413 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:43.268413    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-767bdb48d9-xhf4p"
Sep 15 21:27:43.268894 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:43.268447    2575 scope.go:117] "RemoveContainer" containerID="cc16be1fa494e5bcd5218e0d022f5e065170213d9487b8378c30a62c55d745f1"
Sep 15 21:27:43.284828 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:43.284805    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-767bdb48d9-xhf4p"]
Sep 15 21:27:43.288378 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:43.288359    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-767bdb48d9-xhf4p"]
Sep 15 21:27:44.746480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:44.746447    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab84fc33-7f81-4825-9766-80fb6487e80e" path="/var/lib/kubelet/pods/ab84fc33-7f81-4825-9766-80fb6487e80e/volumes"
Sep 15 21:27:45.102229 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:45.102203    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rsbdn"
Sep 15 21:27:52.353447 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:52.353416    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k2lzl_18d9cbb1-4fad-45f1-ae92-4125ca00a324/serve-healthcheck-canary/0.log"
Sep 15 21:27:53.542161 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.542116    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8dcdf8955-425x9" podUID="b95a4f87-5e3b-47ce-85e1-7e5513324690" containerName="console" containerID="cri-o://b4fd7b4be0cb492ee8a69cfa02dbe39ecd0d2a171f11c38ded1bd6ba48f092d1" gracePeriod=15
Sep 15 21:27:53.832917 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.822696    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:53.894298 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.894261    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-oauth-config\") pod \"b95a4f87-5e3b-47ce-85e1-7e5513324690\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") "
Sep 15 21:27:53.894298 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.894303    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-serving-cert\") pod \"b95a4f87-5e3b-47ce-85e1-7e5513324690\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") "
Sep 15 21:27:53.894541 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.894341    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw5lh\" (UniqueName: \"kubernetes.io/projected/b95a4f87-5e3b-47ce-85e1-7e5513324690-kube-api-access-sw5lh\") pod \"b95a4f87-5e3b-47ce-85e1-7e5513324690\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") "
Sep 15 21:27:53.894541 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.894380    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-service-ca\") pod \"b95a4f87-5e3b-47ce-85e1-7e5513324690\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") "
Sep 15 21:27:53.894541 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.894452    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-config\") pod \"b95a4f87-5e3b-47ce-85e1-7e5513324690\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") "
Sep 15 21:27:53.894541 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.894477    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-oauth-serving-cert\") pod \"b95a4f87-5e3b-47ce-85e1-7e5513324690\" (UID: \"b95a4f87-5e3b-47ce-85e1-7e5513324690\") "
Sep 15 21:27:53.894872 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.894833    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-config" (OuterVolumeSpecName: "console-config") pod "b95a4f87-5e3b-47ce-85e1-7e5513324690" (UID: "b95a4f87-5e3b-47ce-85e1-7e5513324690"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:27:53.894954 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.894878    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-service-ca" (OuterVolumeSpecName: "service-ca") pod "b95a4f87-5e3b-47ce-85e1-7e5513324690" (UID: "b95a4f87-5e3b-47ce-85e1-7e5513324690"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:27:53.894954 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.894890    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b95a4f87-5e3b-47ce-85e1-7e5513324690" (UID: "b95a4f87-5e3b-47ce-85e1-7e5513324690"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:27:53.896659 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.896631    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b95a4f87-5e3b-47ce-85e1-7e5513324690" (UID: "b95a4f87-5e3b-47ce-85e1-7e5513324690"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:27:53.897095 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.897074    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b95a4f87-5e3b-47ce-85e1-7e5513324690" (UID: "b95a4f87-5e3b-47ce-85e1-7e5513324690"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:27:53.897202 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.897119    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b95a4f87-5e3b-47ce-85e1-7e5513324690-kube-api-access-sw5lh" (OuterVolumeSpecName: "kube-api-access-sw5lh") pod "b95a4f87-5e3b-47ce-85e1-7e5513324690" (UID: "b95a4f87-5e3b-47ce-85e1-7e5513324690"). InnerVolumeSpecName "kube-api-access-sw5lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:27:53.996094 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.996064    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-config\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:27:53.996094 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.996095    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-oauth-serving-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:27:53.996274 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.996105    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-oauth-config\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:27:53.996274 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.996113    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b95a4f87-5e3b-47ce-85e1-7e5513324690-console-serving-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:27:53.996274 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.996121    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw5lh\" (UniqueName: \"kubernetes.io/projected/b95a4f87-5e3b-47ce-85e1-7e5513324690-kube-api-access-sw5lh\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:27:53.996274 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:53.996130    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b95a4f87-5e3b-47ce-85e1-7e5513324690-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:27:54.314241 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:54.314207    2575 generic.go:356] "Generic (PLEG): container finished" podID="b95a4f87-5e3b-47ce-85e1-7e5513324690" containerID="b4fd7b4be0cb492ee8a69cfa02dbe39ecd0d2a171f11c38ded1bd6ba48f092d1" exitCode=0
Sep 15 21:27:54.314434 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:54.314250    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8dcdf8955-425x9"
Sep 15 21:27:54.314434 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:54.314279    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dcdf8955-425x9" event={"ID":"b95a4f87-5e3b-47ce-85e1-7e5513324690","Type":"ContainerDied","Data":"b4fd7b4be0cb492ee8a69cfa02dbe39ecd0d2a171f11c38ded1bd6ba48f092d1"}
Sep 15 21:27:54.314434 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:54.314328    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8dcdf8955-425x9" event={"ID":"b95a4f87-5e3b-47ce-85e1-7e5513324690","Type":"ContainerDied","Data":"4ff46e4088741d3cc90b81ede92ff274cdfde008a3dfaa088d772b8f2cad0651"}
Sep 15 21:27:54.314434 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:54.314353    2575 scope.go:117] "RemoveContainer" containerID="b4fd7b4be0cb492ee8a69cfa02dbe39ecd0d2a171f11c38ded1bd6ba48f092d1"
Sep 15 21:27:54.323239 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:54.323221    2575 scope.go:117] "RemoveContainer" containerID="b4fd7b4be0cb492ee8a69cfa02dbe39ecd0d2a171f11c38ded1bd6ba48f092d1"
Sep 15 21:27:54.323507 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:27:54.323490    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4fd7b4be0cb492ee8a69cfa02dbe39ecd0d2a171f11c38ded1bd6ba48f092d1\": container with ID starting with b4fd7b4be0cb492ee8a69cfa02dbe39ecd0d2a171f11c38ded1bd6ba48f092d1 not found: ID does not exist" containerID="b4fd7b4be0cb492ee8a69cfa02dbe39ecd0d2a171f11c38ded1bd6ba48f092d1"
Sep 15 21:27:54.323560 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:54.323516    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4fd7b4be0cb492ee8a69cfa02dbe39ecd0d2a171f11c38ded1bd6ba48f092d1"} err="failed to get container status \"b4fd7b4be0cb492ee8a69cfa02dbe39ecd0d2a171f11c38ded1bd6ba48f092d1\": rpc error: code = NotFound desc = could not find container \"b4fd7b4be0cb492ee8a69cfa02dbe39ecd0d2a171f11c38ded1bd6ba48f092d1\": container with ID starting with b4fd7b4be0cb492ee8a69cfa02dbe39ecd0d2a171f11c38ded1bd6ba48f092d1 not found: ID does not exist"
Sep 15 21:27:54.336115 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:54.336091    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8dcdf8955-425x9"]
Sep 15 21:27:54.339916 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:54.339893    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8dcdf8955-425x9"]
Sep 15 21:27:54.745378 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:54.745351    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b95a4f87-5e3b-47ce-85e1-7e5513324690" path="/var/lib/kubelet/pods/b95a4f87-5e3b-47ce-85e1-7e5513324690/volumes"
Sep 15 21:27:55.526611 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:55.526585    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:27:55.531035 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:27:55.531014    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6fb596986f-np7bm"
Sep 15 21:28:17.294626 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:17.294578    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:17.315373 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:17.315346    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:17.430661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:17.430627    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:31.007540 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:31.007507    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:28:31.467639 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:31.467573    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="alertmanager" containerID="cri-o://ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2" gracePeriod=120
Sep 15 21:28:31.467639 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:31.467633    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="kube-rbac-proxy-metric" containerID="cri-o://636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310" gracePeriod=120
Sep 15 21:28:31.467937 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:31.467683    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="kube-rbac-proxy-web" containerID="cri-o://68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5" gracePeriod=120
Sep 15 21:28:31.467937 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:31.467702    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="kube-rbac-proxy" containerID="cri-o://d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437" gracePeriod=120
Sep 15 21:28:31.467937 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:31.467708    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="prom-label-proxy" containerID="cri-o://05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5" gracePeriod=120
Sep 15 21:28:31.467937 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:31.467733    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="config-reloader" containerID="cri-o://7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec" gracePeriod=120
Sep 15 21:28:32.475587 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.475557    2575 generic.go:356] "Generic (PLEG): container finished" podID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerID="05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5" exitCode=0
Sep 15 21:28:32.475587 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.475580    2575 generic.go:356] "Generic (PLEG): container finished" podID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerID="d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437" exitCode=0
Sep 15 21:28:32.475587 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.475585    2575 generic.go:356] "Generic (PLEG): container finished" podID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerID="7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec" exitCode=0
Sep 15 21:28:32.475587 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.475591    2575 generic.go:356] "Generic (PLEG): container finished" podID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerID="ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2" exitCode=0
Sep 15 21:28:32.476124 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.475629    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c19acc79-0bea-422e-8671-f00fd04c7f1e","Type":"ContainerDied","Data":"05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5"}
Sep 15 21:28:32.476124 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.475769    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c19acc79-0bea-422e-8671-f00fd04c7f1e","Type":"ContainerDied","Data":"d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437"}
Sep 15 21:28:32.476124 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.475814    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c19acc79-0bea-422e-8671-f00fd04c7f1e","Type":"ContainerDied","Data":"7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec"}
Sep 15 21:28:32.476124 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.475853    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c19acc79-0bea-422e-8671-f00fd04c7f1e","Type":"ContainerDied","Data":"ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2"}
Sep 15 21:28:32.712264 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.712243    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:32.829307 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.829202    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-config-volume\") pod \"c19acc79-0bea-422e-8671-f00fd04c7f1e\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") "
Sep 15 21:28:32.829307 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.829263    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy\") pod \"c19acc79-0bea-422e-8671-f00fd04c7f1e\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") "
Sep 15 21:28:32.829307 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.829291    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c19acc79-0bea-422e-8671-f00fd04c7f1e-config-out\") pod \"c19acc79-0bea-422e-8671-f00fd04c7f1e\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") "
Sep 15 21:28:32.829575 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.829335    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c19acc79-0bea-422e-8671-f00fd04c7f1e-metrics-client-ca\") pod \"c19acc79-0bea-422e-8671-f00fd04c7f1e\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") "
Sep 15 21:28:32.829575 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.829370    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy-web\") pod \"c19acc79-0bea-422e-8671-f00fd04c7f1e\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") "
Sep 15 21:28:32.829575 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.829398    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-web-config\") pod \"c19acc79-0bea-422e-8671-f00fd04c7f1e\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") "
Sep 15 21:28:32.829575 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.829517    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c19acc79-0bea-422e-8671-f00fd04c7f1e-tls-assets\") pod \"c19acc79-0bea-422e-8671-f00fd04c7f1e\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") "
Sep 15 21:28:32.829575 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.829566    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbdsk\" (UniqueName: \"kubernetes.io/projected/c19acc79-0bea-422e-8671-f00fd04c7f1e-kube-api-access-xbdsk\") pod \"c19acc79-0bea-422e-8671-f00fd04c7f1e\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") "
Sep 15 21:28:32.829875 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.829601    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c19acc79-0bea-422e-8671-f00fd04c7f1e-alertmanager-main-db\") pod \"c19acc79-0bea-422e-8671-f00fd04c7f1e\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") "
Sep 15 21:28:32.829875 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.829631    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-cluster-tls-config\") pod \"c19acc79-0bea-422e-8671-f00fd04c7f1e\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") "
Sep 15 21:28:32.829875 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.829679    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c19acc79-0bea-422e-8671-f00fd04c7f1e\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") "
Sep 15 21:28:32.829875 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.829728    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-main-tls\") pod \"c19acc79-0bea-422e-8671-f00fd04c7f1e\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") "
Sep 15 21:28:32.829875 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.829801    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c19acc79-0bea-422e-8671-f00fd04c7f1e-alertmanager-trusted-ca-bundle\") pod \"c19acc79-0bea-422e-8671-f00fd04c7f1e\" (UID: \"c19acc79-0bea-422e-8671-f00fd04c7f1e\") "
Sep 15 21:28:32.829875 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.829815    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19acc79-0bea-422e-8671-f00fd04c7f1e-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c19acc79-0bea-422e-8671-f00fd04c7f1e" (UID: "c19acc79-0bea-422e-8671-f00fd04c7f1e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:28:32.830317 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.830225    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c19acc79-0bea-422e-8671-f00fd04c7f1e-metrics-client-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:32.830596 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.830571    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19acc79-0bea-422e-8671-f00fd04c7f1e-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c19acc79-0bea-422e-8671-f00fd04c7f1e" (UID: "c19acc79-0bea-422e-8671-f00fd04c7f1e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:28:32.832688 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.832658    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19acc79-0bea-422e-8671-f00fd04c7f1e-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c19acc79-0bea-422e-8671-f00fd04c7f1e" (UID: "c19acc79-0bea-422e-8671-f00fd04c7f1e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:28:32.832822 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.832737    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19acc79-0bea-422e-8671-f00fd04c7f1e-config-out" (OuterVolumeSpecName: "config-out") pod "c19acc79-0bea-422e-8671-f00fd04c7f1e" (UID: "c19acc79-0bea-422e-8671-f00fd04c7f1e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:28:32.832822 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.832807    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-config-volume" (OuterVolumeSpecName: "config-volume") pod "c19acc79-0bea-422e-8671-f00fd04c7f1e" (UID: "c19acc79-0bea-422e-8671-f00fd04c7f1e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:32.833457 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.833152    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c19acc79-0bea-422e-8671-f00fd04c7f1e" (UID: "c19acc79-0bea-422e-8671-f00fd04c7f1e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:32.833457 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.833369    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19acc79-0bea-422e-8671-f00fd04c7f1e-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c19acc79-0bea-422e-8671-f00fd04c7f1e" (UID: "c19acc79-0bea-422e-8671-f00fd04c7f1e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:28:32.834146 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.834118    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c19acc79-0bea-422e-8671-f00fd04c7f1e" (UID: "c19acc79-0bea-422e-8671-f00fd04c7f1e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:32.834243 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.834193    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c19acc79-0bea-422e-8671-f00fd04c7f1e" (UID: "c19acc79-0bea-422e-8671-f00fd04c7f1e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:32.834348 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.834330    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c19acc79-0bea-422e-8671-f00fd04c7f1e" (UID: "c19acc79-0bea-422e-8671-f00fd04c7f1e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:32.834424 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.834403    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19acc79-0bea-422e-8671-f00fd04c7f1e-kube-api-access-xbdsk" (OuterVolumeSpecName: "kube-api-access-xbdsk") pod "c19acc79-0bea-422e-8671-f00fd04c7f1e" (UID: "c19acc79-0bea-422e-8671-f00fd04c7f1e"). InnerVolumeSpecName "kube-api-access-xbdsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:28:32.837663 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.837576    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c19acc79-0bea-422e-8671-f00fd04c7f1e" (UID: "c19acc79-0bea-422e-8671-f00fd04c7f1e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:32.843565 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.843545    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-web-config" (OuterVolumeSpecName: "web-config") pod "c19acc79-0bea-422e-8671-f00fd04c7f1e" (UID: "c19acc79-0bea-422e-8671-f00fd04c7f1e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:32.931008 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.930977    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:32.931008 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.931008    2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-web-config\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:32.931165 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.931025    2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c19acc79-0bea-422e-8671-f00fd04c7f1e-tls-assets\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:32.931165 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.931041    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbdsk\" (UniqueName: \"kubernetes.io/projected/c19acc79-0bea-422e-8671-f00fd04c7f1e-kube-api-access-xbdsk\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:32.931165 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.931056    2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c19acc79-0bea-422e-8671-f00fd04c7f1e-alertmanager-main-db\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:32.931165 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.931071    2575 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-cluster-tls-config\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:32.931165 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.931085    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:32.931165 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.931096    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-main-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:32.931165 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.931104    2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c19acc79-0bea-422e-8671-f00fd04c7f1e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:32.931165 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.931112    2575 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:32.931165 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.931119    2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c19acc79-0bea-422e-8671-f00fd04c7f1e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:32.931165 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:32.931128    2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c19acc79-0bea-422e-8671-f00fd04c7f1e-config-out\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:33.482368 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.482333    2575 generic.go:356] "Generic (PLEG): container finished" podID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerID="636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310" exitCode=0
Sep 15 21:28:33.482368 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.482359    2575 generic.go:356] "Generic (PLEG): container finished" podID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerID="68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5" exitCode=0
Sep 15 21:28:33.482917 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.482397    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c19acc79-0bea-422e-8671-f00fd04c7f1e","Type":"ContainerDied","Data":"636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310"}
Sep 15 21:28:33.482917 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.482410    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.482917 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.482453    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c19acc79-0bea-422e-8671-f00fd04c7f1e","Type":"ContainerDied","Data":"68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5"}
Sep 15 21:28:33.482917 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.482479    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c19acc79-0bea-422e-8671-f00fd04c7f1e","Type":"ContainerDied","Data":"707521a4900559334fe25cca648ea699928124ea4bd78c87b13db21c49b68877"}
Sep 15 21:28:33.482917 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.482506    2575 scope.go:117] "RemoveContainer" containerID="05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5"
Sep 15 21:28:33.490396 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.490371    2575 scope.go:117] "RemoveContainer" containerID="636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310"
Sep 15 21:28:33.501334 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.501294    2575 scope.go:117] "RemoveContainer" containerID="d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437"
Sep 15 21:28:33.506376 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.506353    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:28:33.510003 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.509985    2575 scope.go:117] "RemoveContainer" containerID="68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5"
Sep 15 21:28:33.510691 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.510666    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:28:33.516631 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.516614    2575 scope.go:117] "RemoveContainer" containerID="7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec"
Sep 15 21:28:33.523114 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.523097    2575 scope.go:117] "RemoveContainer" containerID="ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2"
Sep 15 21:28:33.530239 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.530223    2575 scope.go:117] "RemoveContainer" containerID="4f3cc0b7caeb1229d1e9995418d17053dce3e7e73cb0cff12fba1a3fe6dfb30c"
Sep 15 21:28:33.536691 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.536667    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:28:33.537215 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537194    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="init-config-reloader"
Sep 15 21:28:33.537284 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537218    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="init-config-reloader"
Sep 15 21:28:33.537284 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537241    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab84fc33-7f81-4825-9766-80fb6487e80e" containerName="console"
Sep 15 21:28:33.537284 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537250    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab84fc33-7f81-4825-9766-80fb6487e80e" containerName="console"
Sep 15 21:28:33.537284 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537273    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="kube-rbac-proxy-web"
Sep 15 21:28:33.537284 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537282    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="kube-rbac-proxy-web"
Sep 15 21:28:33.537480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537293    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="kube-rbac-proxy-metric"
Sep 15 21:28:33.537480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537308    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="kube-rbac-proxy-metric"
Sep 15 21:28:33.537480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537325    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="alertmanager"
Sep 15 21:28:33.537480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537333    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="alertmanager"
Sep 15 21:28:33.537480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537342    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="kube-rbac-proxy"
Sep 15 21:28:33.537480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537350    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="kube-rbac-proxy"
Sep 15 21:28:33.537480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537363    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b95a4f87-5e3b-47ce-85e1-7e5513324690" containerName="console"
Sep 15 21:28:33.537480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537371    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b95a4f87-5e3b-47ce-85e1-7e5513324690" containerName="console"
Sep 15 21:28:33.537480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537381    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="prom-label-proxy"
Sep 15 21:28:33.537480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537390    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="prom-label-proxy"
Sep 15 21:28:33.537480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537400    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="config-reloader"
Sep 15 21:28:33.537480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537408    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="config-reloader"
Sep 15 21:28:33.537480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537472    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab84fc33-7f81-4825-9766-80fb6487e80e" containerName="console"
Sep 15 21:28:33.537480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537483    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="alertmanager"
Sep 15 21:28:33.537915 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537494    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b95a4f87-5e3b-47ce-85e1-7e5513324690" containerName="console"
Sep 15 21:28:33.537915 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537503    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="config-reloader"
Sep 15 21:28:33.537915 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537513    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="kube-rbac-proxy"
Sep 15 21:28:33.537915 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537521    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="prom-label-proxy"
Sep 15 21:28:33.537915 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537530    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="kube-rbac-proxy-web"
Sep 15 21:28:33.537915 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.537540    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" containerName="kube-rbac-proxy-metric"
Sep 15 21:28:33.538374 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.538352    2575 scope.go:117] "RemoveContainer" containerID="05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5"
Sep 15 21:28:33.538671 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:28:33.538651    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5\": container with ID starting with 05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5 not found: ID does not exist" containerID="05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5"
Sep 15 21:28:33.538747 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.538679    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5"} err="failed to get container status \"05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5\": rpc error: code = NotFound desc = could not find container \"05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5\": container with ID starting with 05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5 not found: ID does not exist"
Sep 15 21:28:33.538747 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.538698    2575 scope.go:117] "RemoveContainer" containerID="636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310"
Sep 15 21:28:33.539022 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:28:33.538988    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310\": container with ID starting with 636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310 not found: ID does not exist" containerID="636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310"
Sep 15 21:28:33.539084 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.539032    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310"} err="failed to get container status \"636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310\": rpc error: code = NotFound desc = could not find container \"636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310\": container with ID starting with 636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310 not found: ID does not exist"
Sep 15 21:28:33.539084 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.539054    2575 scope.go:117] "RemoveContainer" containerID="d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437"
Sep 15 21:28:33.539324 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:28:33.539306    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437\": container with ID starting with d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437 not found: ID does not exist" containerID="d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437"
Sep 15 21:28:33.539381 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.539327    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437"} err="failed to get container status \"d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437\": rpc error: code = NotFound desc = could not find container \"d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437\": container with ID starting with d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437 not found: ID does not exist"
Sep 15 21:28:33.539381 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.539345    2575 scope.go:117] "RemoveContainer" containerID="68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5"
Sep 15 21:28:33.539609 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:28:33.539587    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5\": container with ID starting with 68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5 not found: ID does not exist" containerID="68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5"
Sep 15 21:28:33.539644 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.539618    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5"} err="failed to get container status \"68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5\": rpc error: code = NotFound desc = could not find container \"68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5\": container with ID starting with 68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5 not found: ID does not exist"
Sep 15 21:28:33.539644 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.539638    2575 scope.go:117] "RemoveContainer" containerID="7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec"
Sep 15 21:28:33.539980 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:28:33.539942    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec\": container with ID starting with 7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec not found: ID does not exist" containerID="7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec"
Sep 15 21:28:33.540062 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.539992    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec"} err="failed to get container status \"7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec\": rpc error: code = NotFound desc = could not find container \"7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec\": container with ID starting with 7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec not found: ID does not exist"
Sep 15 21:28:33.540062 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.540014    2575 scope.go:117] "RemoveContainer" containerID="ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2"
Sep 15 21:28:33.540267 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:28:33.540246    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2\": container with ID starting with ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2 not found: ID does not exist" containerID="ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2"
Sep 15 21:28:33.540304 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.540274    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2"} err="failed to get container status \"ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2\": rpc error: code = NotFound desc = could not find container \"ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2\": container with ID starting with ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2 not found: ID does not exist"
Sep 15 21:28:33.540304 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.540293    2575 scope.go:117] "RemoveContainer" containerID="4f3cc0b7caeb1229d1e9995418d17053dce3e7e73cb0cff12fba1a3fe6dfb30c"
Sep 15 21:28:33.540511 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:28:33.540494    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f3cc0b7caeb1229d1e9995418d17053dce3e7e73cb0cff12fba1a3fe6dfb30c\": container with ID starting with 4f3cc0b7caeb1229d1e9995418d17053dce3e7e73cb0cff12fba1a3fe6dfb30c not found: ID does not exist" containerID="4f3cc0b7caeb1229d1e9995418d17053dce3e7e73cb0cff12fba1a3fe6dfb30c"
Sep 15 21:28:33.540572 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.540519    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3cc0b7caeb1229d1e9995418d17053dce3e7e73cb0cff12fba1a3fe6dfb30c"} err="failed to get container status \"4f3cc0b7caeb1229d1e9995418d17053dce3e7e73cb0cff12fba1a3fe6dfb30c\": rpc error: code = NotFound desc = could not find container \"4f3cc0b7caeb1229d1e9995418d17053dce3e7e73cb0cff12fba1a3fe6dfb30c\": container with ID starting with 4f3cc0b7caeb1229d1e9995418d17053dce3e7e73cb0cff12fba1a3fe6dfb30c not found: ID does not exist"
Sep 15 21:28:33.540572 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.540537    2575 scope.go:117] "RemoveContainer" containerID="05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5"
Sep 15 21:28:33.540773 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.540738    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5"} err="failed to get container status \"05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5\": rpc error: code = NotFound desc = could not find container \"05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5\": container with ID starting with 05f68a8298addc0c1fda0d3dd384a54bd590b4a9922e9ec316411a9e16fad3a5 not found: ID does not exist"
Sep 15 21:28:33.540773 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.540774    2575 scope.go:117] "RemoveContainer" containerID="636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310"
Sep 15 21:28:33.540994 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.540976    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310"} err="failed to get container status \"636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310\": rpc error: code = NotFound desc = could not find container \"636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310\": container with ID starting with 636858d86792f3182fcfeaf4156fbdd69912e390334abb1ff47a0e0160390310 not found: ID does not exist"
Sep 15 21:28:33.541060 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.540996    2575 scope.go:117] "RemoveContainer" containerID="d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437"
Sep 15 21:28:33.541233 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.541209    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437"} err="failed to get container status \"d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437\": rpc error: code = NotFound desc = could not find container \"d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437\": container with ID starting with d7ccac6c057009ac41b6487b703ebd853b213ac809c9feda50f7cc3929916437 not found: ID does not exist"
Sep 15 21:28:33.541233 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.541233    2575 scope.go:117] "RemoveContainer" containerID="68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5"
Sep 15 21:28:33.541495 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.541470    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5"} err="failed to get container status \"68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5\": rpc error: code = NotFound desc = could not find container \"68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5\": container with ID starting with 68ee0771e5593882c1f9cadbccfb0e9f343059483cd5d5aa8d390834606c56b5 not found: ID does not exist"
Sep 15 21:28:33.541495 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.541495    2575 scope.go:117] "RemoveContainer" containerID="7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec"
Sep 15 21:28:33.541949 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.541932    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.544012 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.543982    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec"} err="failed to get container status \"7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec\": rpc error: code = NotFound desc = could not find container \"7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec\": container with ID starting with 7a463505687b6c993d9d595b019c38ea9d27d75ea1d12244f6dd620ef78d39ec not found: ID does not exist"
Sep 15 21:28:33.544099 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.544012    2575 scope.go:117] "RemoveContainer" containerID="ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2"
Sep 15 21:28:33.544577 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.544291    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2"} err="failed to get container status \"ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2\": rpc error: code = NotFound desc = could not find container \"ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2\": container with ID starting with ec573904ea4130299ddbe82074c7bdea69934dc4add741c74709e7cc6a2826a2 not found: ID does not exist"
Sep 15 21:28:33.544577 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.544309    2575 scope.go:117] "RemoveContainer" containerID="4f3cc0b7caeb1229d1e9995418d17053dce3e7e73cb0cff12fba1a3fe6dfb30c"
Sep 15 21:28:33.544694 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.544590    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f3cc0b7caeb1229d1e9995418d17053dce3e7e73cb0cff12fba1a3fe6dfb30c"} err="failed to get container status \"4f3cc0b7caeb1229d1e9995418d17053dce3e7e73cb0cff12fba1a3fe6dfb30c\": rpc error: code = NotFound desc = could not find container \"4f3cc0b7caeb1229d1e9995418d17053dce3e7e73cb0cff12fba1a3fe6dfb30c\": container with ID starting with 4f3cc0b7caeb1229d1e9995418d17053dce3e7e73cb0cff12fba1a3fe6dfb30c not found: ID does not exist"
Sep 15 21:28:33.545673 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.545646    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 15 21:28:33.545793 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.545672    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 15 21:28:33.545793 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.545711    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 15 21:28:33.546003 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.545983    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 15 21:28:33.546092 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.546061    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wg5rj\""
Sep 15 21:28:33.546144 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.546121    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 15 21:28:33.546206 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.546189    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 15 21:28:33.546276 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.546259    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 15 21:28:33.546389 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.546377    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 15 21:28:33.550982 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.550954    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 15 21:28:33.554093 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.554045    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:28:33.737892 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.737797    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.737892 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.737842    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5873c6d7-792c-41ac-b353-8cccb5f17a9b-config-out\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.737892 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.737867    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.737892 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.737889    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5873c6d7-792c-41ac-b353-8cccb5f17a9b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.738209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.737923    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5873c6d7-792c-41ac-b353-8cccb5f17a9b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.738209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.737945    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.738209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.737975    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.738209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.738001    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-config-volume\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.738209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.738022    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-web-config\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.738209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.738051    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.738209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.738079    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5873c6d7-792c-41ac-b353-8cccb5f17a9b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.738209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.738103    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlqxp\" (UniqueName: \"kubernetes.io/projected/5873c6d7-792c-41ac-b353-8cccb5f17a9b-kube-api-access-nlqxp\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.738209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.738149    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5873c6d7-792c-41ac-b353-8cccb5f17a9b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.839039 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.838999    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5873c6d7-792c-41ac-b353-8cccb5f17a9b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.839241 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.839059    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.839241 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.839196    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5873c6d7-792c-41ac-b353-8cccb5f17a9b-config-out\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.839241 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.839234    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.839411 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.839258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5873c6d7-792c-41ac-b353-8cccb5f17a9b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.839411 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.839329    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5873c6d7-792c-41ac-b353-8cccb5f17a9b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.839411 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.839363    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.839607 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.839417    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.839607 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.839453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-config-volume\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.839607 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.839479    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-web-config\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.839607 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.839516    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.839607 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.839561    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5873c6d7-792c-41ac-b353-8cccb5f17a9b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.839607 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.839586    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlqxp\" (UniqueName: \"kubernetes.io/projected/5873c6d7-792c-41ac-b353-8cccb5f17a9b-kube-api-access-nlqxp\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.839928 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.839885    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5873c6d7-792c-41ac-b353-8cccb5f17a9b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.840130 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.840101    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5873c6d7-792c-41ac-b353-8cccb5f17a9b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.842246 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.842220    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5873c6d7-792c-41ac-b353-8cccb5f17a9b-config-out\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.842381 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.842355    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.842494 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.842370    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.843075 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.842728    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.843075 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.842866    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5873c6d7-792c-41ac-b353-8cccb5f17a9b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.843075 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.842913    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-config-volume\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.843075 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.843036    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5873c6d7-792c-41ac-b353-8cccb5f17a9b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.843307 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.843133    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-web-config\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.843431 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.843415    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.844244 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.844228    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5873c6d7-792c-41ac-b353-8cccb5f17a9b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.848493 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.848471    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlqxp\" (UniqueName: \"kubernetes.io/projected/5873c6d7-792c-41ac-b353-8cccb5f17a9b-kube-api-access-nlqxp\") pod \"alertmanager-main-0\" (UID: \"5873c6d7-792c-41ac-b353-8cccb5f17a9b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:33.858371 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:33.858354    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 15 21:28:34.138501 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:34.138478    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 15 21:28:34.140238 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:28:34.140209    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5873c6d7_792c_41ac_b353_8cccb5f17a9b.slice/crio-72795ad89eebecb75f5c345bcd8f78c3fe1f94ca3331cdb47f28d46be068c2b4 WatchSource:0}: Error finding container 72795ad89eebecb75f5c345bcd8f78c3fe1f94ca3331cdb47f28d46be068c2b4: Status 404 returned error can't find the container with id 72795ad89eebecb75f5c345bcd8f78c3fe1f94ca3331cdb47f28d46be068c2b4
Sep 15 21:28:34.488718 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:34.488645    2575 generic.go:356] "Generic (PLEG): container finished" podID="5873c6d7-792c-41ac-b353-8cccb5f17a9b" containerID="a380eebb30a078b398e8c184c4b06b79700368d370a396381696fcfa0c3df73f" exitCode=0
Sep 15 21:28:34.488718 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:34.488691    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5873c6d7-792c-41ac-b353-8cccb5f17a9b","Type":"ContainerDied","Data":"a380eebb30a078b398e8c184c4b06b79700368d370a396381696fcfa0c3df73f"}
Sep 15 21:28:34.489091 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:34.488741    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5873c6d7-792c-41ac-b353-8cccb5f17a9b","Type":"ContainerStarted","Data":"72795ad89eebecb75f5c345bcd8f78c3fe1f94ca3331cdb47f28d46be068c2b4"}
Sep 15 21:28:34.746982 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:34.746960    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c19acc79-0bea-422e-8671-f00fd04c7f1e" path="/var/lib/kubelet/pods/c19acc79-0bea-422e-8671-f00fd04c7f1e/volumes"
Sep 15 21:28:35.036072 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.035991    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-695955f48-mh7hp"]
Sep 15 21:28:35.039398 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.039375    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.041995 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.041968    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 15 21:28:35.041995 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.041986    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 15 21:28:35.042166 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.041999    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 15 21:28:35.042166 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.042015    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 15 21:28:35.042166 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.042112    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-gpbbk\""
Sep 15 21:28:35.042446 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.042431    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 15 21:28:35.047012 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.046990    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 15 21:28:35.047280 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.047252    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ce56fc8-7775-4792-ade7-ed91b0c31571-metrics-client-ca\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.047399 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.047296    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ce56fc8-7775-4792-ade7-ed91b0c31571-telemeter-trusted-ca-bundle\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.047399 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.047357    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ce56fc8-7775-4792-ade7-ed91b0c31571-serving-certs-ca-bundle\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.047399 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.047381    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3ce56fc8-7775-4792-ade7-ed91b0c31571-secret-telemeter-client\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.047569 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.047415    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4cz2\" (UniqueName: \"kubernetes.io/projected/3ce56fc8-7775-4792-ade7-ed91b0c31571-kube-api-access-v4cz2\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.047569 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.047446    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3ce56fc8-7775-4792-ade7-ed91b0c31571-telemeter-client-tls\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.047569 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.047522    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3ce56fc8-7775-4792-ade7-ed91b0c31571-federate-client-tls\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.047866 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.047571    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ce56fc8-7775-4792-ade7-ed91b0c31571-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.053496 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.053470    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-695955f48-mh7hp"]
Sep 15 21:28:35.148474 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.148438    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ce56fc8-7775-4792-ade7-ed91b0c31571-metrics-client-ca\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.148474 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.148486    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ce56fc8-7775-4792-ade7-ed91b0c31571-telemeter-trusted-ca-bundle\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.148720 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.148527    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ce56fc8-7775-4792-ade7-ed91b0c31571-serving-certs-ca-bundle\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.148720 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.148543    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3ce56fc8-7775-4792-ade7-ed91b0c31571-secret-telemeter-client\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.148720 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.148565    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4cz2\" (UniqueName: \"kubernetes.io/projected/3ce56fc8-7775-4792-ade7-ed91b0c31571-kube-api-access-v4cz2\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.148720 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.148590    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3ce56fc8-7775-4792-ade7-ed91b0c31571-telemeter-client-tls\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.148720 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.148618    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3ce56fc8-7775-4792-ade7-ed91b0c31571-federate-client-tls\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.148720 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.148657    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ce56fc8-7775-4792-ade7-ed91b0c31571-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.149317 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.149286    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ce56fc8-7775-4792-ade7-ed91b0c31571-serving-certs-ca-bundle\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.149427 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.149287    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ce56fc8-7775-4792-ade7-ed91b0c31571-metrics-client-ca\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.149427 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.149411    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ce56fc8-7775-4792-ade7-ed91b0c31571-telemeter-trusted-ca-bundle\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.151244 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.151217    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ce56fc8-7775-4792-ade7-ed91b0c31571-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.151511 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.151470    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3ce56fc8-7775-4792-ade7-ed91b0c31571-secret-telemeter-client\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.151511 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.151478    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3ce56fc8-7775-4792-ade7-ed91b0c31571-telemeter-client-tls\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.151730 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.151711    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3ce56fc8-7775-4792-ade7-ed91b0c31571-federate-client-tls\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.157597 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.157574    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4cz2\" (UniqueName: \"kubernetes.io/projected/3ce56fc8-7775-4792-ade7-ed91b0c31571-kube-api-access-v4cz2\") pod \"telemeter-client-695955f48-mh7hp\" (UID: \"3ce56fc8-7775-4792-ade7-ed91b0c31571\") " pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.312296 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.312216    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:28:35.312549 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.312520    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="prometheus" containerID="cri-o://0611236b679062a770c8e119b041fe53b5bb5df68e8cf2cc5ac8c3d9eee71647" gracePeriod=600
Sep 15 21:28:35.312549 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.312545    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="kube-rbac-proxy-web" containerID="cri-o://42cdd07feb6f6d7536e16e3649b7c249fedf721391aec10ae845b834f143cee0" gracePeriod=600
Sep 15 21:28:35.312693 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.312572    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="thanos-sidecar" containerID="cri-o://ebbfbaf07d92fc33075d97e2a55c3faf5a728058ad32f8a60b6f6fe5d3295277" gracePeriod=600
Sep 15 21:28:35.312693 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.312600    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="config-reloader" containerID="cri-o://b75426f6767bc0cf82e7eb54852f7adcbd5bcbc89a9cb6b0ee017facaa22145e" gracePeriod=600
Sep 15 21:28:35.312693 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.312520    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="kube-rbac-proxy-thanos" containerID="cri-o://0d6bd888acb6729a983c9b3786cbab68f6e1b39c4daa39d179ed8f27fc6ee337" gracePeriod=600
Sep 15 21:28:35.312693 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.312572    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="kube-rbac-proxy" containerID="cri-o://cdad8bb0cb9786f1d1d824816b5c6661dd44477ff7de9dbae82a787a961e73f8" gracePeriod=600
Sep 15 21:28:35.350265 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.350230    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-695955f48-mh7hp"
Sep 15 21:28:35.503186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.502707    2575 generic.go:356] "Generic (PLEG): container finished" podID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerID="0d6bd888acb6729a983c9b3786cbab68f6e1b39c4daa39d179ed8f27fc6ee337" exitCode=0
Sep 15 21:28:35.503186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.502735    2575 generic.go:356] "Generic (PLEG): container finished" podID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerID="cdad8bb0cb9786f1d1d824816b5c6661dd44477ff7de9dbae82a787a961e73f8" exitCode=0
Sep 15 21:28:35.503186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.502747    2575 generic.go:356] "Generic (PLEG): container finished" podID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerID="42cdd07feb6f6d7536e16e3649b7c249fedf721391aec10ae845b834f143cee0" exitCode=0
Sep 15 21:28:35.503186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.502774    2575 generic.go:356] "Generic (PLEG): container finished" podID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerID="ebbfbaf07d92fc33075d97e2a55c3faf5a728058ad32f8a60b6f6fe5d3295277" exitCode=0
Sep 15 21:28:35.503186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.502782    2575 generic.go:356] "Generic (PLEG): container finished" podID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerID="b75426f6767bc0cf82e7eb54852f7adcbd5bcbc89a9cb6b0ee017facaa22145e" exitCode=0
Sep 15 21:28:35.503186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.502789    2575 generic.go:356] "Generic (PLEG): container finished" podID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerID="0611236b679062a770c8e119b041fe53b5bb5df68e8cf2cc5ac8c3d9eee71647" exitCode=0
Sep 15 21:28:35.503186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.502886    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df0511db-812d-4853-b5d3-810ad2b50fb2","Type":"ContainerDied","Data":"0d6bd888acb6729a983c9b3786cbab68f6e1b39c4daa39d179ed8f27fc6ee337"}
Sep 15 21:28:35.503186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.502961    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df0511db-812d-4853-b5d3-810ad2b50fb2","Type":"ContainerDied","Data":"cdad8bb0cb9786f1d1d824816b5c6661dd44477ff7de9dbae82a787a961e73f8"}
Sep 15 21:28:35.503186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.503002    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df0511db-812d-4853-b5d3-810ad2b50fb2","Type":"ContainerDied","Data":"42cdd07feb6f6d7536e16e3649b7c249fedf721391aec10ae845b834f143cee0"}
Sep 15 21:28:35.503186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.503043    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df0511db-812d-4853-b5d3-810ad2b50fb2","Type":"ContainerDied","Data":"ebbfbaf07d92fc33075d97e2a55c3faf5a728058ad32f8a60b6f6fe5d3295277"}
Sep 15 21:28:35.503186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.503083    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df0511db-812d-4853-b5d3-810ad2b50fb2","Type":"ContainerDied","Data":"b75426f6767bc0cf82e7eb54852f7adcbd5bcbc89a9cb6b0ee017facaa22145e"}
Sep 15 21:28:35.503186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.503121    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df0511db-812d-4853-b5d3-810ad2b50fb2","Type":"ContainerDied","Data":"0611236b679062a770c8e119b041fe53b5bb5df68e8cf2cc5ac8c3d9eee71647"}
Sep 15 21:28:35.508153 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.508125    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5873c6d7-792c-41ac-b353-8cccb5f17a9b","Type":"ContainerStarted","Data":"15c8bd1001898a349fae1228657f271772a959545c9137ee063748110639c508"}
Sep 15 21:28:35.508294 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.508199    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5873c6d7-792c-41ac-b353-8cccb5f17a9b","Type":"ContainerStarted","Data":"81190fc2a4c2c5fd7cecfecd1d053bc4d7cabe8d2d8a1d8525d8d782e61a9003"}
Sep 15 21:28:35.508294 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.508245    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5873c6d7-792c-41ac-b353-8cccb5f17a9b","Type":"ContainerStarted","Data":"ab9239d4c6e601bbae5920f8e330bc5380557885659305c428f6ed011f19eb06"}
Sep 15 21:28:35.508414 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.508329    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5873c6d7-792c-41ac-b353-8cccb5f17a9b","Type":"ContainerStarted","Data":"87aac684cbb22abfc17990d230afb0b7c7816e4e644c3452cd3f6a8d3a3e0a03"}
Sep 15 21:28:35.508414 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.508376    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5873c6d7-792c-41ac-b353-8cccb5f17a9b","Type":"ContainerStarted","Data":"ecb5be2a6f3d8115185f5cea266e8453362d947d6595aca614ea9dc38b2fbf1f"}
Sep 15 21:28:35.508502 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.508419    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5873c6d7-792c-41ac-b353-8cccb5f17a9b","Type":"ContainerStarted","Data":"f68849ab28af56485021cf2464297e0379574ad95dfecb381bb5200d223d3a5b"}
Sep 15 21:28:35.536487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.536440    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.521678036 podStartE2EDuration="2.53642411s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.746074ms" isStatefulPod=false podCreationTimestamp="2026-09-15 21:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:28:35.5358434 +0000 UTC m=+153.347851842" watchObservedRunningTime="2026-09-15 21:28:35.53642411 +0000 UTC m=+153.348432547"
Sep 15 21:28:35.591608 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.591584    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:35.635792 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.635343    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-695955f48-mh7hp"]
Sep 15 21:28:35.753520 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.753489    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-1\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.753724 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.753529    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-metrics-client-ca\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.753724 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.753561    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df0511db-812d-4853-b5d3-810ad2b50fb2-config-out\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.753724 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.753596    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.753724 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.753629    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.753724 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.753656    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-thanos-prometheus-http-client-file\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.753724 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.753683    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-metrics-client-certs\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.753724 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.753724    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwgzl\" (UniqueName: \"kubernetes.io/projected/df0511db-812d-4853-b5d3-810ad2b50fb2-kube-api-access-pwgzl\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.754101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.753780    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-tls\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.754101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.753808    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-kube-rbac-proxy\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.754101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.753852    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-trusted-ca-bundle\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.754101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.753877    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-2\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.754101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.753913    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-config\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.754101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.753939    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-serving-certs-ca-bundle\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.754101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.753971    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-0\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.754101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.754013    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-kubelet-serving-ca-bundle\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.754101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.754038    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df0511db-812d-4853-b5d3-810ad2b50fb2-tls-assets\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.754101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.754072    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-grpc-tls\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.754101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.754096    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-db\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.754596 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.754118    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-web-config\") pod \"df0511db-812d-4853-b5d3-810ad2b50fb2\" (UID: \"df0511db-812d-4853-b5d3-810ad2b50fb2\") "
Sep 15 21:28:35.755228 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.753903    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:28:35.755331 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.754270    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:28:35.755331 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.754652    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:28:35.755331 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.755202    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:28:35.755331 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.755299    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:28:35.755698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.755673    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:28:35.756644 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.756616    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:28:35.756805 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.756779    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:28:35.757817 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.757777    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:35.758176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.758058    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:35.758176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.758148    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:35.758581 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.758545    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:35.758806 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.758780    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:35.758949 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.758782    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0511db-812d-4853-b5d3-810ad2b50fb2-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:28:35.758949 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.758942    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df0511db-812d-4853-b5d3-810ad2b50fb2-config-out" (OuterVolumeSpecName: "config-out") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:28:35.759077 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.758990    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:35.759192 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.759177    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:35.759453 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.759433    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df0511db-812d-4853-b5d3-810ad2b50fb2-kube-api-access-pwgzl" (OuterVolumeSpecName: "kube-api-access-pwgzl") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "kube-api-access-pwgzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:28:35.759729 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.759715    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-config" (OuterVolumeSpecName: "config") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:35.768558 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.768539    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-web-config" (OuterVolumeSpecName: "web-config") pod "df0511db-812d-4853-b5d3-810ad2b50fb2" (UID: "df0511db-812d-4853-b5d3-810ad2b50fb2"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:28:35.854724 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854688    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854724 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854719    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-metrics-client-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854734    2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df0511db-812d-4853-b5d3-810ad2b50fb2-config-out\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854748    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854779    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854792    2575 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-thanos-prometheus-http-client-file\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854803    2575 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-metrics-client-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854815    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwgzl\" (UniqueName: \"kubernetes.io/projected/df0511db-812d-4853-b5d3-810ad2b50fb2-kube-api-access-pwgzl\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854830    2575 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-prometheus-k8s-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854843    2575 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-kube-rbac-proxy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854855    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-trusted-ca-bundle\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854866    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854878    2575 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-config\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854890    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854905    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854917    2575 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df0511db-812d-4853-b5d3-810ad2b50fb2-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854930    2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df0511db-812d-4853-b5d3-810ad2b50fb2-tls-assets\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854942    2575 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-secret-grpc-tls\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854954    2575 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/df0511db-812d-4853-b5d3-810ad2b50fb2-prometheus-k8s-db\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:35.854977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:35.854966    2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df0511db-812d-4853-b5d3-810ad2b50fb2-web-config\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:28:36.514560 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.514527    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-695955f48-mh7hp" event={"ID":"3ce56fc8-7775-4792-ade7-ed91b0c31571","Type":"ContainerStarted","Data":"9abf8bd7dbdd5238d3cf4c6c3e4c50606747f9f14b414bfa88bd1c5a29ff513f"}
Sep 15 21:28:36.517486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.517469    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.517616 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.517481    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"df0511db-812d-4853-b5d3-810ad2b50fb2","Type":"ContainerDied","Data":"fa5bcb312f6ed0340b02992e15f653fe8efdc987ad5091a615f88b98dd001700"}
Sep 15 21:28:36.517616 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.517546    2575 scope.go:117] "RemoveContainer" containerID="0d6bd888acb6729a983c9b3786cbab68f6e1b39c4daa39d179ed8f27fc6ee337"
Sep 15 21:28:36.525254 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.525231    2575 scope.go:117] "RemoveContainer" containerID="cdad8bb0cb9786f1d1d824816b5c6661dd44477ff7de9dbae82a787a961e73f8"
Sep 15 21:28:36.533507 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.533401    2575 scope.go:117] "RemoveContainer" containerID="42cdd07feb6f6d7536e16e3649b7c249fedf721391aec10ae845b834f143cee0"
Sep 15 21:28:36.541014 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.540986    2575 scope.go:117] "RemoveContainer" containerID="ebbfbaf07d92fc33075d97e2a55c3faf5a728058ad32f8a60b6f6fe5d3295277"
Sep 15 21:28:36.545227 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.545205    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:28:36.549550 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.549530    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:28:36.550197 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.550177    2575 scope.go:117] "RemoveContainer" containerID="b75426f6767bc0cf82e7eb54852f7adcbd5bcbc89a9cb6b0ee017facaa22145e"
Sep 15 21:28:36.556720 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.556703    2575 scope.go:117] "RemoveContainer" containerID="0611236b679062a770c8e119b041fe53b5bb5df68e8cf2cc5ac8c3d9eee71647"
Sep 15 21:28:36.563936 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.563882    2575 scope.go:117] "RemoveContainer" containerID="5b78d669ad6a69ddf69f2fed14b35228b8bd815f4900c812aaafcec4507cb8fa"
Sep 15 21:28:36.572898 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.572879    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:28:36.573190 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573178    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="thanos-sidecar"
Sep 15 21:28:36.573225 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573192    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="thanos-sidecar"
Sep 15 21:28:36.573225 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573202    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="prometheus"
Sep 15 21:28:36.573225 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573207    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="prometheus"
Sep 15 21:28:36.573225 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573215    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="init-config-reloader"
Sep 15 21:28:36.573225 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573221    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="init-config-reloader"
Sep 15 21:28:36.573361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573229    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="config-reloader"
Sep 15 21:28:36.573361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573234    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="config-reloader"
Sep 15 21:28:36.573361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573248    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="kube-rbac-proxy-web"
Sep 15 21:28:36.573361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573253    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="kube-rbac-proxy-web"
Sep 15 21:28:36.573361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573263    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="kube-rbac-proxy"
Sep 15 21:28:36.573361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573268    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="kube-rbac-proxy"
Sep 15 21:28:36.573361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573275    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="kube-rbac-proxy-thanos"
Sep 15 21:28:36.573361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573280    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="kube-rbac-proxy-thanos"
Sep 15 21:28:36.573361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573322    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="thanos-sidecar"
Sep 15 21:28:36.573361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573328    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="kube-rbac-proxy-web"
Sep 15 21:28:36.573361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573336    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="kube-rbac-proxy-thanos"
Sep 15 21:28:36.573361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573342    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="config-reloader"
Sep 15 21:28:36.573361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573346    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="prometheus"
Sep 15 21:28:36.573361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.573352    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" containerName="kube-rbac-proxy"
Sep 15 21:28:36.578520 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.578504    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.582596 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.582560    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 15 21:28:36.582695 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.582569    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 15 21:28:36.582695 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.582616    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 15 21:28:36.582831 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.582734    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 15 21:28:36.582831 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.582783    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 15 21:28:36.582928 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.582834    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 15 21:28:36.583132 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.582851    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 15 21:28:36.583178 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.582930    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 15 21:28:36.583236 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.582938    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 15 21:28:36.583236 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.582946    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 15 21:28:36.583236 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.583214    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9jlkd9tn1go9h\""
Sep 15 21:28:36.583367 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.583039    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dcjrj\""
Sep 15 21:28:36.583367 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.583260    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 15 21:28:36.583712 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.583696    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 15 21:28:36.584941 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.584921    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 15 21:28:36.586812 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.586736    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 15 21:28:36.593614 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.593591    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:28:36.661280 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661280 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661282    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661504 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661308    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661504 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661340    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661504 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661367    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661504 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661395    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661504 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661420    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661504 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661447    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661504 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661476    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661504 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661501    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661543    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661610    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661644    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pj54\" (UniqueName: \"kubernetes.io/projected/a5675c14-383b-4a3a-ad51-5481b7d38c06-kube-api-access-4pj54\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661679    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-config\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661710    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5675c14-383b-4a3a-ad51-5481b7d38c06-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661733    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5675c14-383b-4a3a-ad51-5481b7d38c06-config-out\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661801    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661829    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661887    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-web-config\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.661930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.661914    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a5675c14-383b-4a3a-ad51-5481b7d38c06-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.746910 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.746878    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df0511db-812d-4853-b5d3-810ad2b50fb2" path="/var/lib/kubelet/pods/df0511db-812d-4853-b5d3-810ad2b50fb2/volumes"
Sep 15 21:28:36.762780 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.762744    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.762839 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.762786    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.762839 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.762804    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.762839 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.762826    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.762941 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.762850    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.762941 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.762873    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.762941 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.762899    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.762941 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.762928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pj54\" (UniqueName: \"kubernetes.io/projected/a5675c14-383b-4a3a-ad51-5481b7d38c06-kube-api-access-4pj54\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.763121 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.762956    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-config\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.763121 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.762976    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5675c14-383b-4a3a-ad51-5481b7d38c06-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.763121 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.762996    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5675c14-383b-4a3a-ad51-5481b7d38c06-config-out\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.763121 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.763025    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.763121 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.763050    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.763121 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.763077    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-web-config\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.763121 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.763099    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a5675c14-383b-4a3a-ad51-5481b7d38c06-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.763455 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.763140    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.763455 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.763164    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.763455 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.763198    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.763455 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.763225    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.763455 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.763261    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.763455 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.763436    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.763919 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.763728    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.763919 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.763908    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.764136 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.764109    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.764987 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.764927    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.766402 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.766144    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.766402 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.766169    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.766402 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.766254    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.766402 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.766299    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.766644 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.766408    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.766644 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.766569    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a5675c14-383b-4a3a-ad51-5481b7d38c06-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.767308 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.767232    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.767795 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.767770    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.768625 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.768605    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.768717 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.768620    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-config\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.768877 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.768851    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a5675c14-383b-4a3a-ad51-5481b7d38c06-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.768990 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.768972    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a5675c14-383b-4a3a-ad51-5481b7d38c06-web-config\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.769062 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.769048    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a5675c14-383b-4a3a-ad51-5481b7d38c06-config-out\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.769127 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.769104    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a5675c14-383b-4a3a-ad51-5481b7d38c06-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.775374 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.775350    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pj54\" (UniqueName: \"kubernetes.io/projected/a5675c14-383b-4a3a-ad51-5481b7d38c06-kube-api-access-4pj54\") pod \"prometheus-k8s-0\" (UID: \"a5675c14-383b-4a3a-ad51-5481b7d38c06\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:36.889826 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:36.889786    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:28:37.166925 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:37.166802    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 15 21:28:37.169961 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:28:37.169928    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5675c14_383b_4a3a_ad51_5481b7d38c06.slice/crio-b24e4d985df00b7c234b0803f0671f724420e0e8b7bdefa16ae170436f3e99f9 WatchSource:0}: Error finding container b24e4d985df00b7c234b0803f0671f724420e0e8b7bdefa16ae170436f3e99f9: Status 404 returned error can't find the container with id b24e4d985df00b7c234b0803f0671f724420e0e8b7bdefa16ae170436f3e99f9
Sep 15 21:28:37.525960 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:37.525870    2575 generic.go:356] "Generic (PLEG): container finished" podID="a5675c14-383b-4a3a-ad51-5481b7d38c06" containerID="04a9a70faf63ceebf7c1ad11fd3ed324b6b021d136e9f62d3fc977f6fa9f9794" exitCode=0
Sep 15 21:28:37.526352 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:37.525958    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5675c14-383b-4a3a-ad51-5481b7d38c06","Type":"ContainerDied","Data":"04a9a70faf63ceebf7c1ad11fd3ed324b6b021d136e9f62d3fc977f6fa9f9794"}
Sep 15 21:28:37.526352 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:37.526022    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5675c14-383b-4a3a-ad51-5481b7d38c06","Type":"ContainerStarted","Data":"b24e4d985df00b7c234b0803f0671f724420e0e8b7bdefa16ae170436f3e99f9"}
Sep 15 21:28:38.533938 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:38.533904    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5675c14-383b-4a3a-ad51-5481b7d38c06","Type":"ContainerStarted","Data":"ad3fb53608e060fc6a6f3c8550ce36dc02a44975e91d2869f54767e9e25271cb"}
Sep 15 21:28:38.534320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:38.534024    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5675c14-383b-4a3a-ad51-5481b7d38c06","Type":"ContainerStarted","Data":"157eac03851e986a916ca18f4bd1ccdb4fd31d3768927bcd5a2cebb6ae8eb84d"}
Sep 15 21:28:38.534320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:38.534054    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5675c14-383b-4a3a-ad51-5481b7d38c06","Type":"ContainerStarted","Data":"b72f204ede2995ce18a0666e30f6312233cb8772dddd3b6855ca2d4159ac87de"}
Sep 15 21:28:38.534320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:38.534078    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5675c14-383b-4a3a-ad51-5481b7d38c06","Type":"ContainerStarted","Data":"e6d4bd1c438dfb946d2f13878d5e38eb0a8e021363d8c4db0cf6a3e2159eb5e1"}
Sep 15 21:28:38.534320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:38.534109    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5675c14-383b-4a3a-ad51-5481b7d38c06","Type":"ContainerStarted","Data":"aa407df01d1e48edaee062956e6da7b785a31a186ed5bdde45d1c1f2199564a7"}
Sep 15 21:28:38.534320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:38.534145    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a5675c14-383b-4a3a-ad51-5481b7d38c06","Type":"ContainerStarted","Data":"3b46bbd36d3a5d1f78dcebf553edac4796f1bc9aa52a2d1fe574f80614fe30bb"}
Sep 15 21:28:38.535950 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:38.535923    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-695955f48-mh7hp" event={"ID":"3ce56fc8-7775-4792-ade7-ed91b0c31571","Type":"ContainerStarted","Data":"35c949f758a6347e909fef323297cb1185e03b81988915da4741b79bba325d5b"}
Sep 15 21:28:38.536045 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:38.535974    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-695955f48-mh7hp" event={"ID":"3ce56fc8-7775-4792-ade7-ed91b0c31571","Type":"ContainerStarted","Data":"e9e38bbafd99d876dd8329b151aa0981f582283b8742ccca1a9ba8ed0176f46e"}
Sep 15 21:28:38.536045 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:38.536002    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-695955f48-mh7hp" event={"ID":"3ce56fc8-7775-4792-ade7-ed91b0c31571","Type":"ContainerStarted","Data":"2eb9253688a0636f27c0cb2da1a4a21a3238eea26961bd7323ec650e014d3f82"}
Sep 15 21:28:38.561908 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:38.561866    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.539587246 podStartE2EDuration="2.561850421s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.263175ms" isStatefulPod=false podCreationTimestamp="2026-09-15 21:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:28:38.560135007 +0000 UTC m=+156.372143450" watchObservedRunningTime="2026-09-15 21:28:38.561850421 +0000 UTC m=+156.373858863"
Sep 15 21:28:38.581679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:38.581540    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-695955f48-mh7hp" podStartSLOduration=1.360906365 podStartE2EDuration="3.581520426s" totalImagesPullingTime="2.220614061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:28:38.581055561 +0000 UTC m=+156.393064014" watchObservedRunningTime="2026-09-15 21:28:38.581520426 +0000 UTC m=+156.393528946"
Sep 15 21:28:41.890732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:28:41.890685    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:29:36.890394 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:29:36.890354    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:29:36.905984 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:29:36.905956    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:29:37.787113 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:29:37.787083    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 15 21:31:02.622411 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:02.622386    2575 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 21:31:14.092048 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.092015    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mbf94"]
Sep 15 21:31:14.095291 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.095276    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mbf94"
Sep 15 21:31:14.097531 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.097506    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-tzjdz\""
Sep 15 21:31:14.097674 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.097568    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 15 21:31:14.098169 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.098151    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 15 21:31:14.103904 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.103882    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mbf94"]
Sep 15 21:31:14.210178 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.210145    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w548\" (UniqueName: \"kubernetes.io/projected/badf7617-9655-4702-81cc-637d8cfe0c66-kube-api-access-7w548\") pod \"cert-manager-webhook-6486ffd997-mbf94\" (UID: \"badf7617-9655-4702-81cc-637d8cfe0c66\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mbf94"
Sep 15 21:31:14.210350 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.210207    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/badf7617-9655-4702-81cc-637d8cfe0c66-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mbf94\" (UID: \"badf7617-9655-4702-81cc-637d8cfe0c66\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mbf94"
Sep 15 21:31:14.310913 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.310867    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w548\" (UniqueName: \"kubernetes.io/projected/badf7617-9655-4702-81cc-637d8cfe0c66-kube-api-access-7w548\") pod \"cert-manager-webhook-6486ffd997-mbf94\" (UID: \"badf7617-9655-4702-81cc-637d8cfe0c66\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mbf94"
Sep 15 21:31:14.311057 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.310948    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/badf7617-9655-4702-81cc-637d8cfe0c66-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mbf94\" (UID: \"badf7617-9655-4702-81cc-637d8cfe0c66\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mbf94"
Sep 15 21:31:14.318709 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.318683    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w548\" (UniqueName: \"kubernetes.io/projected/badf7617-9655-4702-81cc-637d8cfe0c66-kube-api-access-7w548\") pod \"cert-manager-webhook-6486ffd997-mbf94\" (UID: \"badf7617-9655-4702-81cc-637d8cfe0c66\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mbf94"
Sep 15 21:31:14.318891 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.318873    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/badf7617-9655-4702-81cc-637d8cfe0c66-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-mbf94\" (UID: \"badf7617-9655-4702-81cc-637d8cfe0c66\") " pod="cert-manager/cert-manager-webhook-6486ffd997-mbf94"
Sep 15 21:31:14.377692 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.377662    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hbngz"]
Sep 15 21:31:14.380973 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.380956    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hbngz"
Sep 15 21:31:14.382900 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.382881    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-j2qv8\""
Sep 15 21:31:14.389743 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.389723    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hbngz"]
Sep 15 21:31:14.413031 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.412999    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-mbf94"
Sep 15 21:31:14.513456 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.513414    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/458aef53-f14c-45fe-825b-85fbf00aad34-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hbngz\" (UID: \"458aef53-f14c-45fe-825b-85fbf00aad34\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hbngz"
Sep 15 21:31:14.513639 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.513465    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqn6c\" (UniqueName: \"kubernetes.io/projected/458aef53-f14c-45fe-825b-85fbf00aad34-kube-api-access-sqn6c\") pod \"cert-manager-cainjector-559cdd8c87-hbngz\" (UID: \"458aef53-f14c-45fe-825b-85fbf00aad34\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hbngz"
Sep 15 21:31:14.614179 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.614133    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/458aef53-f14c-45fe-825b-85fbf00aad34-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hbngz\" (UID: \"458aef53-f14c-45fe-825b-85fbf00aad34\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hbngz"
Sep 15 21:31:14.614378 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.614189    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqn6c\" (UniqueName: \"kubernetes.io/projected/458aef53-f14c-45fe-825b-85fbf00aad34-kube-api-access-sqn6c\") pod \"cert-manager-cainjector-559cdd8c87-hbngz\" (UID: \"458aef53-f14c-45fe-825b-85fbf00aad34\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hbngz"
Sep 15 21:31:14.621924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.621896    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/458aef53-f14c-45fe-825b-85fbf00aad34-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-hbngz\" (UID: \"458aef53-f14c-45fe-825b-85fbf00aad34\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hbngz"
Sep 15 21:31:14.622043 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.622024    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqn6c\" (UniqueName: \"kubernetes.io/projected/458aef53-f14c-45fe-825b-85fbf00aad34-kube-api-access-sqn6c\") pod \"cert-manager-cainjector-559cdd8c87-hbngz\" (UID: \"458aef53-f14c-45fe-825b-85fbf00aad34\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-hbngz"
Sep 15 21:31:14.679909 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.679793    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-mbf94"]
Sep 15 21:31:14.682774 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:31:14.682725    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbadf7617_9655_4702_81cc_637d8cfe0c66.slice/crio-e32bf08c291bf89e6b1cfcd7f1f1d7ac2d792170abcae082424add5de293914e WatchSource:0}: Error finding container e32bf08c291bf89e6b1cfcd7f1f1d7ac2d792170abcae082424add5de293914e: Status 404 returned error can't find the container with id e32bf08c291bf89e6b1cfcd7f1f1d7ac2d792170abcae082424add5de293914e
Sep 15 21:31:14.684534 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.684521    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:31:14.690176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.690156    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hbngz"
Sep 15 21:31:14.959974 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:14.959950    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-hbngz"]
Sep 15 21:31:14.962360 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:31:14.962336    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod458aef53_f14c_45fe_825b_85fbf00aad34.slice/crio-fcebf7a928189acaf5614c2f2de2238db7bc1431f35136f144b24d01affd1890 WatchSource:0}: Error finding container fcebf7a928189acaf5614c2f2de2238db7bc1431f35136f144b24d01affd1890: Status 404 returned error can't find the container with id fcebf7a928189acaf5614c2f2de2238db7bc1431f35136f144b24d01affd1890
Sep 15 21:31:15.145482 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:15.145450    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hbngz" event={"ID":"458aef53-f14c-45fe-825b-85fbf00aad34","Type":"ContainerStarted","Data":"fcebf7a928189acaf5614c2f2de2238db7bc1431f35136f144b24d01affd1890"}
Sep 15 21:31:15.146640 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:15.146612    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mbf94" event={"ID":"badf7617-9655-4702-81cc-637d8cfe0c66","Type":"ContainerStarted","Data":"e32bf08c291bf89e6b1cfcd7f1f1d7ac2d792170abcae082424add5de293914e"}
Sep 15 21:31:19.167488 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:19.167448    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hbngz" event={"ID":"458aef53-f14c-45fe-825b-85fbf00aad34","Type":"ContainerStarted","Data":"438db5accc333913f5e193219fb2997f60eb233c42c8f49c62556a367fdb68a3"}
Sep 15 21:31:19.169175 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:19.169151    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-mbf94" event={"ID":"badf7617-9655-4702-81cc-637d8cfe0c66","Type":"ContainerStarted","Data":"44ec74356986955729633da64975d277b213cf1c516db0d1452f756dfca17910"}
Sep 15 21:31:19.169288 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:19.169209    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mbf94"
Sep 15 21:31:19.210392 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:19.210350    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-hbngz" podStartSLOduration=2.015245993 podStartE2EDuration="5.210335748s" totalImagesPullingTime="3.195089755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:19.208431225 +0000 UTC m=+317.020439667" watchObservedRunningTime="2026-09-15 21:31:19.210335748 +0000 UTC m=+317.022344190"
Sep 15 21:31:19.236373 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:19.236333    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-mbf94" podStartSLOduration=1.7518354569999999 podStartE2EDuration="5.236318267s" totalImagesPullingTime="3.48448281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:19.234992457 +0000 UTC m=+317.047000914" watchObservedRunningTime="2026-09-15 21:31:19.236318267 +0000 UTC m=+317.048326710"
Sep 15 21:31:25.175348 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:25.175320    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-mbf94"
Sep 15 21:31:45.516097 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.516062    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"]
Sep 15 21:31:45.522959 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.522931    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:45.525042 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.525021    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 21:31:45.525188 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.525078    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 21:31:45.525188 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.525104    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-5f75r\""
Sep 15 21:31:45.525422 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.525402    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 21:31:45.525473 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.525426    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 21:31:45.537942 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.537921    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"]
Sep 15 21:31:45.571477 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.571449    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855-tmp\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-qzxzg\" (UID: \"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:45.571578 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.571488    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkhkr\" (UniqueName: \"kubernetes.io/projected/8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855-kube-api-access-wkhkr\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-qzxzg\" (UID: \"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:45.571578 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.571518    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-qzxzg\" (UID: \"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:45.571578 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.571562    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855-manifests\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-qzxzg\" (UID: \"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:45.571679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.571588    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855-webhook-cert\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-qzxzg\" (UID: \"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:45.672254 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.672222    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855-tmp\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-qzxzg\" (UID: \"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:45.672407 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.672266    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkhkr\" (UniqueName: \"kubernetes.io/projected/8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855-kube-api-access-wkhkr\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-qzxzg\" (UID: \"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:45.672407 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.672384    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-qzxzg\" (UID: \"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:45.672524 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.672456    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855-manifests\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-qzxzg\" (UID: \"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:45.672524 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.672491    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855-webhook-cert\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-qzxzg\" (UID: \"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:45.672788 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.672743    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855-manifests\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-qzxzg\" (UID: \"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:45.674560 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.674534    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855-tmp\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-qzxzg\" (UID: \"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:45.674779 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.674738    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855-webhook-cert\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-qzxzg\" (UID: \"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:45.674941 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.674924    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855-apiservice-cert\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-qzxzg\" (UID: \"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:45.684733 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.684715    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkhkr\" (UniqueName: \"kubernetes.io/projected/8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855-kube-api-access-wkhkr\") pod \"opendatahub-operator-controller-manager-77fddcc9c6-qzxzg\" (UID: \"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855\") " pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:45.834079 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:45.834009    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:46.115228 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:46.115197    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"]
Sep 15 21:31:46.118346 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:31:46.118319    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca1bf6f_b8cd_4cc2_a0fd_b8a2f067d855.slice/crio-9c666fd60354e1ee7272f68b32fcc6771cf26ac2641583cb35bf685007ec8838 WatchSource:0}: Error finding container 9c666fd60354e1ee7272f68b32fcc6771cf26ac2641583cb35bf685007ec8838: Status 404 returned error can't find the container with id 9c666fd60354e1ee7272f68b32fcc6771cf26ac2641583cb35bf685007ec8838
Sep 15 21:31:46.272515 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:46.272481    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg" event={"ID":"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855","Type":"ContainerStarted","Data":"9c666fd60354e1ee7272f68b32fcc6771cf26ac2641583cb35bf685007ec8838"}
Sep 15 21:31:51.296733 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:51.296692    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg" event={"ID":"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855","Type":"ContainerStarted","Data":"2d3ec5cd7c9004e105bd2eb8aff10e0390917320eb805754182b992751957e25"}
Sep 15 21:31:52.302541 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:52.302510    2575 generic.go:356] "Generic (PLEG): container finished" podID="8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855" containerID="2d3ec5cd7c9004e105bd2eb8aff10e0390917320eb805754182b992751957e25" exitCode=0
Sep 15 21:31:52.302940 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:52.302599    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg" event={"ID":"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855","Type":"ContainerDied","Data":"2d3ec5cd7c9004e105bd2eb8aff10e0390917320eb805754182b992751957e25"}
Sep 15 21:31:53.308459 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:53.308423    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg" event={"ID":"8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855","Type":"ContainerStarted","Data":"c6ee2c673998728c779280f4f6128a3bc7fa0439f6505c3a883ebdc539606a79"}
Sep 15 21:31:53.308870 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:53.308552    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:31:53.329842 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:31:53.329795    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg" podStartSLOduration=2.451835595 podStartE2EDuration="8.329779486s" totalImagesPullingTime="4.816948899s" totalInitContainerRuntime="1.060994992s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:31:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:31:53.328084836 +0000 UTC m=+351.140093294" watchObservedRunningTime="2026-09-15 21:31:53.329779486 +0000 UTC m=+351.141787923"
Sep 15 21:32:04.315434 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:04.315400    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-77fddcc9c6-qzxzg"
Sep 15 21:32:06.225935 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.225895    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"]
Sep 15 21:32:06.236290 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.236261    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"
Sep 15 21:32:06.240146 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.240123    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 15 21:32:06.241351 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.241325    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-brf94\""
Sep 15 21:32:06.246315 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.241571    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 15 21:32:06.246315 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.241611    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 21:32:06.246315 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.241620    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 21:32:06.246315 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.241778    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 15 21:32:06.246315 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.246307    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"]
Sep 15 21:32:06.343798 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.343737    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eac363c9-9f81-4e65-abb6-3ed190030f39-metrics-cert\") pod \"lws-controller-manager-5d697dcf85-gtz98\" (UID: \"eac363c9-9f81-4e65-abb6-3ed190030f39\") " pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"
Sep 15 21:32:06.343985 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.343819    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eac363c9-9f81-4e65-abb6-3ed190030f39-manager-config\") pod \"lws-controller-manager-5d697dcf85-gtz98\" (UID: \"eac363c9-9f81-4e65-abb6-3ed190030f39\") " pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"
Sep 15 21:32:06.343985 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.343871    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d6lv\" (UniqueName: \"kubernetes.io/projected/eac363c9-9f81-4e65-abb6-3ed190030f39-kube-api-access-2d6lv\") pod \"lws-controller-manager-5d697dcf85-gtz98\" (UID: \"eac363c9-9f81-4e65-abb6-3ed190030f39\") " pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"
Sep 15 21:32:06.343985 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.343951    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eac363c9-9f81-4e65-abb6-3ed190030f39-cert\") pod \"lws-controller-manager-5d697dcf85-gtz98\" (UID: \"eac363c9-9f81-4e65-abb6-3ed190030f39\") " pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"
Sep 15 21:32:06.445074 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.445031    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eac363c9-9f81-4e65-abb6-3ed190030f39-manager-config\") pod \"lws-controller-manager-5d697dcf85-gtz98\" (UID: \"eac363c9-9f81-4e65-abb6-3ed190030f39\") " pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"
Sep 15 21:32:06.445266 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.445082    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d6lv\" (UniqueName: \"kubernetes.io/projected/eac363c9-9f81-4e65-abb6-3ed190030f39-kube-api-access-2d6lv\") pod \"lws-controller-manager-5d697dcf85-gtz98\" (UID: \"eac363c9-9f81-4e65-abb6-3ed190030f39\") " pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"
Sep 15 21:32:06.445266 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.445157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eac363c9-9f81-4e65-abb6-3ed190030f39-cert\") pod \"lws-controller-manager-5d697dcf85-gtz98\" (UID: \"eac363c9-9f81-4e65-abb6-3ed190030f39\") " pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"
Sep 15 21:32:06.445389 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.445279    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eac363c9-9f81-4e65-abb6-3ed190030f39-metrics-cert\") pod \"lws-controller-manager-5d697dcf85-gtz98\" (UID: \"eac363c9-9f81-4e65-abb6-3ed190030f39\") " pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"
Sep 15 21:32:06.445715 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.445687    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eac363c9-9f81-4e65-abb6-3ed190030f39-manager-config\") pod \"lws-controller-manager-5d697dcf85-gtz98\" (UID: \"eac363c9-9f81-4e65-abb6-3ed190030f39\") " pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"
Sep 15 21:32:06.447693 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.447661    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eac363c9-9f81-4e65-abb6-3ed190030f39-metrics-cert\") pod \"lws-controller-manager-5d697dcf85-gtz98\" (UID: \"eac363c9-9f81-4e65-abb6-3ed190030f39\") " pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"
Sep 15 21:32:06.447821 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.447708    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eac363c9-9f81-4e65-abb6-3ed190030f39-cert\") pod \"lws-controller-manager-5d697dcf85-gtz98\" (UID: \"eac363c9-9f81-4e65-abb6-3ed190030f39\") " pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"
Sep 15 21:32:06.461677 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.461647    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d6lv\" (UniqueName: \"kubernetes.io/projected/eac363c9-9f81-4e65-abb6-3ed190030f39-kube-api-access-2d6lv\") pod \"lws-controller-manager-5d697dcf85-gtz98\" (UID: \"eac363c9-9f81-4e65-abb6-3ed190030f39\") " pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"
Sep 15 21:32:06.551069 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.550983    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"
Sep 15 21:32:06.828278 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:06.828212    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"]
Sep 15 21:32:06.831562 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:32:06.831535    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac363c9_9f81_4e65_abb6_3ed190030f39.slice/crio-d6e1fc4b897c21ed19534f63e653fa7e0b5b7660a78b3ac5246548a3279833a5 WatchSource:0}: Error finding container d6e1fc4b897c21ed19534f63e653fa7e0b5b7660a78b3ac5246548a3279833a5: Status 404 returned error can't find the container with id d6e1fc4b897c21ed19534f63e653fa7e0b5b7660a78b3ac5246548a3279833a5
Sep 15 21:32:07.369183 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:07.369145    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98" event={"ID":"eac363c9-9f81-4e65-abb6-3ed190030f39","Type":"ContainerStarted","Data":"d6e1fc4b897c21ed19534f63e653fa7e0b5b7660a78b3ac5246548a3279833a5"}
Sep 15 21:32:10.384598 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:10.384564    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98" event={"ID":"eac363c9-9f81-4e65-abb6-3ed190030f39","Type":"ContainerStarted","Data":"6496c2b5cfe48ac57d68b3ccf9b3637f44e3caa277a23423b51b5d542f22f7c2"}
Sep 15 21:32:10.384976 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:10.384684    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"
Sep 15 21:32:10.402170 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:10.402122    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98" podStartSLOduration=1.855655171 podStartE2EDuration="4.402108785s" totalImagesPullingTime="2.546453614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:10.401046051 +0000 UTC m=+368.213054496" watchObservedRunningTime="2026-09-15 21:32:10.402108785 +0000 UTC m=+368.214117228"
Sep 15 21:32:16.771424 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.771384    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d78dff5cc-c254b"]
Sep 15 21:32:16.775425 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.775407    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d78dff5cc-c254b"
Sep 15 21:32:16.777839 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.777816    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 21:32:16.777960 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.777852    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 21:32:16.777960 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.777851    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 21:32:16.778089 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.777997    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 21:32:16.778717 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.778702    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rfdsv\""
Sep 15 21:32:16.790000 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.789975    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d78dff5cc-c254b"]
Sep 15 21:32:16.835113 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.835078    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993a3b58-915c-48cf-860e-0695b63089b4-tls-certs\") pod \"kube-auth-proxy-d78dff5cc-c254b\" (UID: \"993a3b58-915c-48cf-860e-0695b63089b4\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-c254b"
Sep 15 21:32:16.835113 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.835109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/993a3b58-915c-48cf-860e-0695b63089b4-tmp\") pod \"kube-auth-proxy-d78dff5cc-c254b\" (UID: \"993a3b58-915c-48cf-860e-0695b63089b4\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-c254b"
Sep 15 21:32:16.835358 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.835216    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzft2\" (UniqueName: \"kubernetes.io/projected/993a3b58-915c-48cf-860e-0695b63089b4-kube-api-access-vzft2\") pod \"kube-auth-proxy-d78dff5cc-c254b\" (UID: \"993a3b58-915c-48cf-860e-0695b63089b4\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-c254b"
Sep 15 21:32:16.936604 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.936565    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993a3b58-915c-48cf-860e-0695b63089b4-tls-certs\") pod \"kube-auth-proxy-d78dff5cc-c254b\" (UID: \"993a3b58-915c-48cf-860e-0695b63089b4\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-c254b"
Sep 15 21:32:16.936604 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.936596    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/993a3b58-915c-48cf-860e-0695b63089b4-tmp\") pod \"kube-auth-proxy-d78dff5cc-c254b\" (UID: \"993a3b58-915c-48cf-860e-0695b63089b4\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-c254b"
Sep 15 21:32:16.936884 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.936649    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzft2\" (UniqueName: \"kubernetes.io/projected/993a3b58-915c-48cf-860e-0695b63089b4-kube-api-access-vzft2\") pod \"kube-auth-proxy-d78dff5cc-c254b\" (UID: \"993a3b58-915c-48cf-860e-0695b63089b4\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-c254b"
Sep 15 21:32:16.938963 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.938935    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/993a3b58-915c-48cf-860e-0695b63089b4-tmp\") pod \"kube-auth-proxy-d78dff5cc-c254b\" (UID: \"993a3b58-915c-48cf-860e-0695b63089b4\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-c254b"
Sep 15 21:32:16.939088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.939067    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/993a3b58-915c-48cf-860e-0695b63089b4-tls-certs\") pod \"kube-auth-proxy-d78dff5cc-c254b\" (UID: \"993a3b58-915c-48cf-860e-0695b63089b4\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-c254b"
Sep 15 21:32:16.943951 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:16.943932    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzft2\" (UniqueName: \"kubernetes.io/projected/993a3b58-915c-48cf-860e-0695b63089b4-kube-api-access-vzft2\") pod \"kube-auth-proxy-d78dff5cc-c254b\" (UID: \"993a3b58-915c-48cf-860e-0695b63089b4\") " pod="openshift-ingress/kube-auth-proxy-d78dff5cc-c254b"
Sep 15 21:32:17.085860 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:17.085771    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d78dff5cc-c254b"
Sep 15 21:32:17.357719 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:17.357679    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d78dff5cc-c254b"]
Sep 15 21:32:17.361473 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:32:17.361445    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993a3b58_915c_48cf_860e_0695b63089b4.slice/crio-dcaebee165d0cd7803b142575efc9b082888d6ef987816801e4d2c92bd69425a WatchSource:0}: Error finding container dcaebee165d0cd7803b142575efc9b082888d6ef987816801e4d2c92bd69425a: Status 404 returned error can't find the container with id dcaebee165d0cd7803b142575efc9b082888d6ef987816801e4d2c92bd69425a
Sep 15 21:32:17.421606 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:17.421559    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d78dff5cc-c254b" event={"ID":"993a3b58-915c-48cf-860e-0695b63089b4","Type":"ContainerStarted","Data":"dcaebee165d0cd7803b142575efc9b082888d6ef987816801e4d2c92bd69425a"}
Sep 15 21:32:20.439829 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:20.439797    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d78dff5cc-c254b" event={"ID":"993a3b58-915c-48cf-860e-0695b63089b4","Type":"ContainerStarted","Data":"94cbf7c7f6343dccd8bd1666e2618dcffc7def53244ba42a7c64191be70dd041"}
Sep 15 21:32:20.456632 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:20.456583    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d78dff5cc-c254b" podStartSLOduration=1.585907306 podStartE2EDuration="4.456567998s" totalImagesPullingTime="2.870660692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:32:20.455500138 +0000 UTC m=+378.267508583" watchObservedRunningTime="2026-09-15 21:32:20.456567998 +0000 UTC m=+378.268576441"
Sep 15 21:32:21.391592 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:32:21.391559    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5d697dcf85-gtz98"
Sep 15 21:33:49.743949 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:49.743913    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw"]
Sep 15 21:33:49.746286 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:49.746270    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw"
Sep 15 21:33:49.748984 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:49.748960    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 21:33:49.749109 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:49.749039    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 21:33:49.749649 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:49.749635    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 15 21:33:49.749700 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:49.749648    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 15 21:33:49.749700 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:49.749682    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-csqbq\""
Sep 15 21:33:49.756101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:49.756078    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw"]
Sep 15 21:33:49.844603 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:49.844563    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99k5s\" (UniqueName: \"kubernetes.io/projected/1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8-kube-api-access-99k5s\") pod \"kuadrant-console-plugin-fbdd4f445-mgcqw\" (UID: \"1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw"
Sep 15 21:33:49.844809 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:49.844620    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-mgcqw\" (UID: \"1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw"
Sep 15 21:33:49.844809 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:49.844659    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-mgcqw\" (UID: \"1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw"
Sep 15 21:33:49.946050 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:49.946004    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-mgcqw\" (UID: \"1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw"
Sep 15 21:33:49.946253 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:49.946071    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-mgcqw\" (UID: \"1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw"
Sep 15 21:33:49.946253 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:49.946218    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99k5s\" (UniqueName: \"kubernetes.io/projected/1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8-kube-api-access-99k5s\") pod \"kuadrant-console-plugin-fbdd4f445-mgcqw\" (UID: \"1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw"
Sep 15 21:33:49.947439 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:49.947411    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-mgcqw\" (UID: \"1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw"
Sep 15 21:33:49.947563 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:33:49.947546    2575 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 15 21:33:49.947637 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:33:49.947627    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8-plugin-serving-cert podName:1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8 nodeName:}" failed. No retries permitted until 2026-09-15 21:33:50.44760698 +0000 UTC m=+468.259615418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-mgcqw" (UID: "1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8") : secret "plugin-serving-cert" not found
Sep 15 21:33:49.958416 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:49.958382    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99k5s\" (UniqueName: \"kubernetes.io/projected/1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8-kube-api-access-99k5s\") pod \"kuadrant-console-plugin-fbdd4f445-mgcqw\" (UID: \"1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw"
Sep 15 21:33:50.451438 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:50.451392    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-mgcqw\" (UID: \"1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw"
Sep 15 21:33:50.454137 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:50.454110    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-mgcqw\" (UID: \"1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw"
Sep 15 21:33:50.656201 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:50.656159    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw"
Sep 15 21:33:50.938247 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:50.938222    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw"]
Sep 15 21:33:50.940972 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:33:50.940944    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a3fa7b2_cf4b_4bf2_bc45_3c157683a3a8.slice/crio-759579cb794833131bff27841b35ad6c8460d4271050eaa6cc8279b880bf7493 WatchSource:0}: Error finding container 759579cb794833131bff27841b35ad6c8460d4271050eaa6cc8279b880bf7493: Status 404 returned error can't find the container with id 759579cb794833131bff27841b35ad6c8460d4271050eaa6cc8279b880bf7493
Sep 15 21:33:51.855831 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:51.855787    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw" event={"ID":"1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8","Type":"ContainerStarted","Data":"759579cb794833131bff27841b35ad6c8460d4271050eaa6cc8279b880bf7493"}
Sep 15 21:33:56.887270 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:56.887236    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw" event={"ID":"1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8","Type":"ContainerStarted","Data":"fa17bf38dbf3d901d0145625141d82d1f0a28cf96ebad394b6438af1efbe88e8"}
Sep 15 21:33:56.902646 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:56.902593    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-mgcqw" podStartSLOduration=2.389400868 podStartE2EDuration="7.902576565s" totalImagesPullingTime="5.513175697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:33:56.901198667 +0000 UTC m=+474.713207125" watchObservedRunningTime="2026-09-15 21:33:56.902576565 +0000 UTC m=+474.714585011"
Sep 15 21:33:59.855980 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:59.855944    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"]
Sep 15 21:33:59.859729 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:59.859714    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"
Sep 15 21:33:59.862075 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:59.862058    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-52mqn\""
Sep 15 21:33:59.873416 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:59.873392    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"]
Sep 15 21:33:59.942040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:59.942018    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/351ef252-ab64-4403-96e3-90e4576d6181-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lkqbf\" (UID: \"351ef252-ab64-4403-96e3-90e4576d6181\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"
Sep 15 21:33:59.942198 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:33:59.942057    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn2g7\" (UniqueName: \"kubernetes.io/projected/351ef252-ab64-4403-96e3-90e4576d6181-kube-api-access-jn2g7\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lkqbf\" (UID: \"351ef252-ab64-4403-96e3-90e4576d6181\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"
Sep 15 21:34:00.043041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.043014    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/351ef252-ab64-4403-96e3-90e4576d6181-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lkqbf\" (UID: \"351ef252-ab64-4403-96e3-90e4576d6181\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"
Sep 15 21:34:00.043151 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.043056    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn2g7\" (UniqueName: \"kubernetes.io/projected/351ef252-ab64-4403-96e3-90e4576d6181-kube-api-access-jn2g7\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lkqbf\" (UID: \"351ef252-ab64-4403-96e3-90e4576d6181\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"
Sep 15 21:34:00.043372 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.043354    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/351ef252-ab64-4403-96e3-90e4576d6181-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lkqbf\" (UID: \"351ef252-ab64-4403-96e3-90e4576d6181\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"
Sep 15 21:34:00.059782 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.059738    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn2g7\" (UniqueName: \"kubernetes.io/projected/351ef252-ab64-4403-96e3-90e4576d6181-kube-api-access-jn2g7\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-lkqbf\" (UID: \"351ef252-ab64-4403-96e3-90e4576d6181\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"
Sep 15 21:34:00.168798 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.168693    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"
Sep 15 21:34:00.231486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.231400    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"]
Sep 15 21:34:00.239891 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.239865    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"]
Sep 15 21:34:00.253921 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.253896    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"]
Sep 15 21:34:00.258355 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.258337    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"
Sep 15 21:34:00.274058 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.274032    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"]
Sep 15 21:34:00.345766 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.345720    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6r95\" (UniqueName: \"kubernetes.io/projected/44b79675-b304-4d45-866c-4ca092859e3f-kube-api-access-g6r95\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dgdmz\" (UID: \"44b79675-b304-4d45-866c-4ca092859e3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"
Sep 15 21:34:00.345919 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.345782    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/44b79675-b304-4d45-866c-4ca092859e3f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dgdmz\" (UID: \"44b79675-b304-4d45-866c-4ca092859e3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"
Sep 15 21:34:00.447446 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.447415    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6r95\" (UniqueName: \"kubernetes.io/projected/44b79675-b304-4d45-866c-4ca092859e3f-kube-api-access-g6r95\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dgdmz\" (UID: \"44b79675-b304-4d45-866c-4ca092859e3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"
Sep 15 21:34:00.447586 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.447460    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/44b79675-b304-4d45-866c-4ca092859e3f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dgdmz\" (UID: \"44b79675-b304-4d45-866c-4ca092859e3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"
Sep 15 21:34:00.447824 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.447807    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/44b79675-b304-4d45-866c-4ca092859e3f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dgdmz\" (UID: \"44b79675-b304-4d45-866c-4ca092859e3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"
Sep 15 21:34:00.462000 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.461968    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6r95\" (UniqueName: \"kubernetes.io/projected/44b79675-b304-4d45-866c-4ca092859e3f-kube-api-access-g6r95\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dgdmz\" (UID: \"44b79675-b304-4d45-866c-4ca092859e3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"
Sep 15 21:34:00.548015 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:34:00.547977    2575 log.go:32] "RunPodSandbox from runtime service failed" err=<
Sep 15 21:34:00.548015 ip-10-0-135-148 kubenswrapper[2575]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-lkqbf_kuadrant-system_351ef252-ab64-4403-96e3-90e4576d6181_0(11d75cd53b6df4151a0bb3043d3c5717f8f2c353345404cfeca770322597b044): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-lkqbf to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"11d75cd53b6df4151a0bb3043d3c5717f8f2c353345404cfeca770322597b044" Netns:"/var/run/netns/c7a4456e-e4ae-4397-8a9a-9e7f66be9a94" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-lkqbf;K8S_POD_INFRA_CONTAINER_ID=11d75cd53b6df4151a0bb3043d3c5717f8f2c353345404cfeca770322597b044;K8S_POD_UID=351ef252-ab64-4403-96e3-90e4576d6181" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf/351ef252-ab64-4403-96e3-90e4576d6181:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf 11d75cd53b6df4151a0bb3043d3c5717f8f2c353345404cfeca770322597b044 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:1c [10.134.0.28/23]
Sep 15 21:34:00.548015 ip-10-0-135-148 kubenswrapper[2575]:         '
Sep 15 21:34:00.548015 ip-10-0-135-148 kubenswrapper[2575]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 15 21:34:00.548015 ip-10-0-135-148 kubenswrapper[2575]:  >
Sep 15 21:34:00.548249 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:34:00.548049    2575 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Sep 15 21:34:00.548249 ip-10-0-135-148 kubenswrapper[2575]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-lkqbf_kuadrant-system_351ef252-ab64-4403-96e3-90e4576d6181_0(11d75cd53b6df4151a0bb3043d3c5717f8f2c353345404cfeca770322597b044): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-lkqbf to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"11d75cd53b6df4151a0bb3043d3c5717f8f2c353345404cfeca770322597b044" Netns:"/var/run/netns/c7a4456e-e4ae-4397-8a9a-9e7f66be9a94" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-lkqbf;K8S_POD_INFRA_CONTAINER_ID=11d75cd53b6df4151a0bb3043d3c5717f8f2c353345404cfeca770322597b044;K8S_POD_UID=351ef252-ab64-4403-96e3-90e4576d6181" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf/351ef252-ab64-4403-96e3-90e4576d6181:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf 11d75cd53b6df4151a0bb3043d3c5717f8f2c353345404cfeca770322597b044 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:1c [10.134.0.28/23]
Sep 15 21:34:00.548249 ip-10-0-135-148 kubenswrapper[2575]:         '
Sep 15 21:34:00.548249 ip-10-0-135-148 kubenswrapper[2575]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 15 21:34:00.548249 ip-10-0-135-148 kubenswrapper[2575]:  > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"
Sep 15 21:34:00.569740 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.569718    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"
Sep 15 21:34:00.847865 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.847843    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"]
Sep 15 21:34:00.850508 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:34:00.850478    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b79675_b304_4d45_866c_4ca092859e3f.slice/crio-04250d1aa088b8636cd0c9af87d0d62afb141bdb52ac1df120d2fa9c81eeead2 WatchSource:0}: Error finding container 04250d1aa088b8636cd0c9af87d0d62afb141bdb52ac1df120d2fa9c81eeead2: Status 404 returned error can't find the container with id 04250d1aa088b8636cd0c9af87d0d62afb141bdb52ac1df120d2fa9c81eeead2
Sep 15 21:34:00.907284 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.907261    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"
Sep 15 21:34:00.907637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.907261    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz" event={"ID":"44b79675-b304-4d45-866c-4ca092859e3f","Type":"ContainerStarted","Data":"04250d1aa088b8636cd0c9af87d0d62afb141bdb52ac1df120d2fa9c81eeead2"}
Sep 15 21:34:00.915210 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:00.915185    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"
Sep 15 21:34:00.917894 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:34:00.917855    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-lkqbf\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="351ef252-ab64-4403-96e3-90e4576d6181" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"
Sep 15 21:34:01.052975 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:01.052943    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn2g7\" (UniqueName: \"kubernetes.io/projected/351ef252-ab64-4403-96e3-90e4576d6181-kube-api-access-jn2g7\") pod \"351ef252-ab64-4403-96e3-90e4576d6181\" (UID: \"351ef252-ab64-4403-96e3-90e4576d6181\") "
Sep 15 21:34:01.053142 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:01.052998    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/351ef252-ab64-4403-96e3-90e4576d6181-extensions-socket-volume\") pod \"351ef252-ab64-4403-96e3-90e4576d6181\" (UID: \"351ef252-ab64-4403-96e3-90e4576d6181\") "
Sep 15 21:34:01.053276 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:01.053251    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351ef252-ab64-4403-96e3-90e4576d6181-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "351ef252-ab64-4403-96e3-90e4576d6181" (UID: "351ef252-ab64-4403-96e3-90e4576d6181"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:34:01.054994 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:01.054973    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351ef252-ab64-4403-96e3-90e4576d6181-kube-api-access-jn2g7" (OuterVolumeSpecName: "kube-api-access-jn2g7") pod "351ef252-ab64-4403-96e3-90e4576d6181" (UID: "351ef252-ab64-4403-96e3-90e4576d6181"). InnerVolumeSpecName "kube-api-access-jn2g7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:34:01.154147 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:01.154069    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn2g7\" (UniqueName: \"kubernetes.io/projected/351ef252-ab64-4403-96e3-90e4576d6181-kube-api-access-jn2g7\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:34:01.154147 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:01.154097    2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/351ef252-ab64-4403-96e3-90e4576d6181-extensions-socket-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:34:01.916464 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:01.916424    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"
Sep 15 21:34:01.918488 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:34:01.918456    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-lkqbf\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="351ef252-ab64-4403-96e3-90e4576d6181" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"
Sep 15 21:34:01.924280 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:34:01.924253    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-lkqbf\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="351ef252-ab64-4403-96e3-90e4576d6181" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"
Sep 15 21:34:02.746415 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:34:02.746380    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-lkqbf\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="351ef252-ab64-4403-96e3-90e4576d6181" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-lkqbf"
Sep 15 21:34:02.749186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:02.749163    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="351ef252-ab64-4403-96e3-90e4576d6181" path="/var/lib/kubelet/pods/351ef252-ab64-4403-96e3-90e4576d6181/volumes"
Sep 15 21:34:12.977093 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:12.977062    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz" event={"ID":"44b79675-b304-4d45-866c-4ca092859e3f","Type":"ContainerStarted","Data":"b4403cc111a7dcd3a0e9cb66bed07d94c73a61b1b5c826ea41c1bf4042af9d7c"}
Sep 15 21:34:12.977462 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:12.977160    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"
Sep 15 21:34:12.996991 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:12.996948    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz" podStartSLOduration=1.004112229 podStartE2EDuration="12.996933918s" totalImagesPullingTime="11.992821689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:34:12.996094142 +0000 UTC m=+490.808102586" watchObservedRunningTime="2026-09-15 21:34:12.996933918 +0000 UTC m=+490.808942361"
Sep 15 21:34:23.985563 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:23.985530    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"
Sep 15 21:34:36.256698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:36.256667    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"]
Sep 15 21:34:36.257168 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:36.256936    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz" podUID="44b79675-b304-4d45-866c-4ca092859e3f" containerName="manager" containerID="cri-o://b4403cc111a7dcd3a0e9cb66bed07d94c73a61b1b5c826ea41c1bf4042af9d7c" gracePeriod=10
Sep 15 21:34:36.506281 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:36.506255    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"
Sep 15 21:34:36.569913 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:36.569830    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6r95\" (UniqueName: \"kubernetes.io/projected/44b79675-b304-4d45-866c-4ca092859e3f-kube-api-access-g6r95\") pod \"44b79675-b304-4d45-866c-4ca092859e3f\" (UID: \"44b79675-b304-4d45-866c-4ca092859e3f\") "
Sep 15 21:34:36.570054 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:36.569943    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/44b79675-b304-4d45-866c-4ca092859e3f-extensions-socket-volume\") pod \"44b79675-b304-4d45-866c-4ca092859e3f\" (UID: \"44b79675-b304-4d45-866c-4ca092859e3f\") "
Sep 15 21:34:36.570395 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:36.570372    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44b79675-b304-4d45-866c-4ca092859e3f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "44b79675-b304-4d45-866c-4ca092859e3f" (UID: "44b79675-b304-4d45-866c-4ca092859e3f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:34:36.572105 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:36.572075    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b79675-b304-4d45-866c-4ca092859e3f-kube-api-access-g6r95" (OuterVolumeSpecName: "kube-api-access-g6r95") pod "44b79675-b304-4d45-866c-4ca092859e3f" (UID: "44b79675-b304-4d45-866c-4ca092859e3f"). InnerVolumeSpecName "kube-api-access-g6r95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:34:36.671374 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:36.671339    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6r95\" (UniqueName: \"kubernetes.io/projected/44b79675-b304-4d45-866c-4ca092859e3f-kube-api-access-g6r95\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:34:36.671374 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:36.671369    2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/44b79675-b304-4d45-866c-4ca092859e3f-extensions-socket-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:34:37.093930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:37.093894    2575 generic.go:356] "Generic (PLEG): container finished" podID="44b79675-b304-4d45-866c-4ca092859e3f" containerID="b4403cc111a7dcd3a0e9cb66bed07d94c73a61b1b5c826ea41c1bf4042af9d7c" exitCode=0
Sep 15 21:34:37.094101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:37.093951    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"
Sep 15 21:34:37.094101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:37.093977    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz" event={"ID":"44b79675-b304-4d45-866c-4ca092859e3f","Type":"ContainerDied","Data":"b4403cc111a7dcd3a0e9cb66bed07d94c73a61b1b5c826ea41c1bf4042af9d7c"}
Sep 15 21:34:37.094101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:37.094028    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz" event={"ID":"44b79675-b304-4d45-866c-4ca092859e3f","Type":"ContainerDied","Data":"04250d1aa088b8636cd0c9af87d0d62afb141bdb52ac1df120d2fa9c81eeead2"}
Sep 15 21:34:37.094101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:37.094050    2575 scope.go:117] "RemoveContainer" containerID="b4403cc111a7dcd3a0e9cb66bed07d94c73a61b1b5c826ea41c1bf4042af9d7c"
Sep 15 21:34:37.102655 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:37.102640    2575 scope.go:117] "RemoveContainer" containerID="b4403cc111a7dcd3a0e9cb66bed07d94c73a61b1b5c826ea41c1bf4042af9d7c"
Sep 15 21:34:37.102949 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:34:37.102930    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4403cc111a7dcd3a0e9cb66bed07d94c73a61b1b5c826ea41c1bf4042af9d7c\": container with ID starting with b4403cc111a7dcd3a0e9cb66bed07d94c73a61b1b5c826ea41c1bf4042af9d7c not found: ID does not exist" containerID="b4403cc111a7dcd3a0e9cb66bed07d94c73a61b1b5c826ea41c1bf4042af9d7c"
Sep 15 21:34:37.103033 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:37.102958    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4403cc111a7dcd3a0e9cb66bed07d94c73a61b1b5c826ea41c1bf4042af9d7c"} err="failed to get container status \"b4403cc111a7dcd3a0e9cb66bed07d94c73a61b1b5c826ea41c1bf4042af9d7c\": rpc error: code = NotFound desc = could not find container \"b4403cc111a7dcd3a0e9cb66bed07d94c73a61b1b5c826ea41c1bf4042af9d7c\": container with ID starting with b4403cc111a7dcd3a0e9cb66bed07d94c73a61b1b5c826ea41c1bf4042af9d7c not found: ID does not exist"
Sep 15 21:34:37.128068 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:37.128042    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"]
Sep 15 21:34:37.149177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:37.149152    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dgdmz"]
Sep 15 21:34:38.746727 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:38.746692    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44b79675-b304-4d45-866c-4ca092859e3f" path="/var/lib/kubelet/pods/44b79675-b304-4d45-866c-4ca092859e3f/volumes"
Sep 15 21:34:43.787631 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:43.787597    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:34:43.788112 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:43.787964    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44b79675-b304-4d45-866c-4ca092859e3f" containerName="manager"
Sep 15 21:34:43.788112 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:43.787977    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="44b79675-b304-4d45-866c-4ca092859e3f" containerName="manager"
Sep 15 21:34:43.788112 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:43.788044    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="44b79675-b304-4d45-866c-4ca092859e3f" containerName="manager"
Sep 15 21:34:43.790565 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:43.790549    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-shtqn"
Sep 15 21:34:43.792575 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:43.792555    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 21:34:43.800191 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:43.800169    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:34:43.826557 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:43.826536    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:34:43.942509 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:43.942477    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26rjq\" (UniqueName: \"kubernetes.io/projected/0841737a-b3ce-4940-8e0a-0e48c5e3d882-kube-api-access-26rjq\") pod \"limitador-limitador-84994f7ffc-shtqn\" (UID: \"0841737a-b3ce-4940-8e0a-0e48c5e3d882\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-shtqn"
Sep 15 21:34:43.942696 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:43.942604    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0841737a-b3ce-4940-8e0a-0e48c5e3d882-config-file\") pod \"limitador-limitador-84994f7ffc-shtqn\" (UID: \"0841737a-b3ce-4940-8e0a-0e48c5e3d882\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-shtqn"
Sep 15 21:34:44.043486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:44.043387    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0841737a-b3ce-4940-8e0a-0e48c5e3d882-config-file\") pod \"limitador-limitador-84994f7ffc-shtqn\" (UID: \"0841737a-b3ce-4940-8e0a-0e48c5e3d882\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-shtqn"
Sep 15 21:34:44.043486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:44.043448    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26rjq\" (UniqueName: \"kubernetes.io/projected/0841737a-b3ce-4940-8e0a-0e48c5e3d882-kube-api-access-26rjq\") pod \"limitador-limitador-84994f7ffc-shtqn\" (UID: \"0841737a-b3ce-4940-8e0a-0e48c5e3d882\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-shtqn"
Sep 15 21:34:44.044048 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:44.044029    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0841737a-b3ce-4940-8e0a-0e48c5e3d882-config-file\") pod \"limitador-limitador-84994f7ffc-shtqn\" (UID: \"0841737a-b3ce-4940-8e0a-0e48c5e3d882\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-shtqn"
Sep 15 21:34:44.051543 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:44.051516    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26rjq\" (UniqueName: \"kubernetes.io/projected/0841737a-b3ce-4940-8e0a-0e48c5e3d882-kube-api-access-26rjq\") pod \"limitador-limitador-84994f7ffc-shtqn\" (UID: \"0841737a-b3ce-4940-8e0a-0e48c5e3d882\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-shtqn"
Sep 15 21:34:44.101256 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:44.101226    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-shtqn"
Sep 15 21:34:44.372215 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:44.372185    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:34:44.375267 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:34:44.375240    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0841737a_b3ce_4940_8e0a_0e48c5e3d882.slice/crio-167cd1c5a1c738bbb8d8fb830b871c8d5ce5ebf56a3c8b3ef23c822f1d1cca46 WatchSource:0}: Error finding container 167cd1c5a1c738bbb8d8fb830b871c8d5ce5ebf56a3c8b3ef23c822f1d1cca46: Status 404 returned error can't find the container with id 167cd1c5a1c738bbb8d8fb830b871c8d5ce5ebf56a3c8b3ef23c822f1d1cca46
Sep 15 21:34:45.134395 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:45.134355    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-shtqn" event={"ID":"0841737a-b3ce-4940-8e0a-0e48c5e3d882","Type":"ContainerStarted","Data":"167cd1c5a1c738bbb8d8fb830b871c8d5ce5ebf56a3c8b3ef23c822f1d1cca46"}
Sep 15 21:34:58.201108 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:58.201067    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-shtqn" event={"ID":"0841737a-b3ce-4940-8e0a-0e48c5e3d882","Type":"ContainerStarted","Data":"8fb86e36c08e1228a44174d67b2dda29d33dcfee6d82dd075e5ff3339aa64908"}
Sep 15 21:34:58.201483 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:58.201140    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-shtqn"
Sep 15 21:34:58.218146 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:34:58.218099    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-shtqn" podStartSLOduration=1.871282129 podStartE2EDuration="15.21808539s" totalImagesPullingTime="13.346803261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:34:58.217168954 +0000 UTC m=+536.029177401" watchObservedRunningTime="2026-09-15 21:34:58.21808539 +0000 UTC m=+536.030093834"
Sep 15 21:35:09.207988 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:35:09.207957    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-shtqn"
Sep 15 21:36:11.323440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:11.323406    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-xx5vp"]
Sep 15 21:36:11.326615 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:11.326600    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xx5vp"
Sep 15 21:36:11.330553 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:11.330531    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-zd78j\""
Sep 15 21:36:11.330769 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:11.330742    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 15 21:36:11.346732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:11.346708    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-xx5vp"]
Sep 15 21:36:11.388999 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:11.388970    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e58dd44-0283-4acd-a0c1-3984a3451a52-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-xx5vp\" (UID: \"2e58dd44-0283-4acd-a0c1-3984a3451a52\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xx5vp"
Sep 15 21:36:11.389181 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:11.389036    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j744g\" (UniqueName: \"kubernetes.io/projected/2e58dd44-0283-4acd-a0c1-3984a3451a52-kube-api-access-j744g\") pod \"payload-pre-processing-59fbc47bf8-xx5vp\" (UID: \"2e58dd44-0283-4acd-a0c1-3984a3451a52\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xx5vp"
Sep 15 21:36:11.400743 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:11.400713    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:36:11.489561 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:11.489530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e58dd44-0283-4acd-a0c1-3984a3451a52-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-xx5vp\" (UID: \"2e58dd44-0283-4acd-a0c1-3984a3451a52\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xx5vp"
Sep 15 21:36:11.489712 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:11.489587    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j744g\" (UniqueName: \"kubernetes.io/projected/2e58dd44-0283-4acd-a0c1-3984a3451a52-kube-api-access-j744g\") pod \"payload-pre-processing-59fbc47bf8-xx5vp\" (UID: \"2e58dd44-0283-4acd-a0c1-3984a3451a52\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xx5vp"
Sep 15 21:36:11.490291 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:11.490273    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e58dd44-0283-4acd-a0c1-3984a3451a52-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-xx5vp\" (UID: \"2e58dd44-0283-4acd-a0c1-3984a3451a52\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xx5vp"
Sep 15 21:36:11.513728 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:11.513700    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j744g\" (UniqueName: \"kubernetes.io/projected/2e58dd44-0283-4acd-a0c1-3984a3451a52-kube-api-access-j744g\") pod \"payload-pre-processing-59fbc47bf8-xx5vp\" (UID: \"2e58dd44-0283-4acd-a0c1-3984a3451a52\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xx5vp"
Sep 15 21:36:11.635881 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:11.635859    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xx5vp"
Sep 15 21:36:11.908479 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:11.908455    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-xx5vp"]
Sep 15 21:36:11.911173 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:36:11.911139    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e58dd44_0283_4acd_a0c1_3984a3451a52.slice/crio-4afe1c29d83f8f5d7df5fb37e42e77914b9274c6ee0973509aa73aa130491c2e WatchSource:0}: Error finding container 4afe1c29d83f8f5d7df5fb37e42e77914b9274c6ee0973509aa73aa130491c2e: Status 404 returned error can't find the container with id 4afe1c29d83f8f5d7df5fb37e42e77914b9274c6ee0973509aa73aa130491c2e
Sep 15 21:36:12.552366 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:12.552332    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xx5vp" event={"ID":"2e58dd44-0283-4acd-a0c1-3984a3451a52","Type":"ContainerStarted","Data":"4afe1c29d83f8f5d7df5fb37e42e77914b9274c6ee0973509aa73aa130491c2e"}
Sep 15 21:36:13.558520 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:13.558441    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xx5vp" event={"ID":"2e58dd44-0283-4acd-a0c1-3984a3451a52","Type":"ContainerStarted","Data":"3121a4cb63901b1cddee1575dbefe83f00cf121ea2a5d6e74a68664d581ef13f"}
Sep 15 21:36:13.558909 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:13.558580    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xx5vp"
Sep 15 21:36:13.572542 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:13.572491    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xx5vp" podStartSLOduration=1.245859544 podStartE2EDuration="2.572478828s" totalImagesPullingTime="1.326619284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:13.571378814 +0000 UTC m=+611.383387259" watchObservedRunningTime="2026-09-15 21:36:13.572478828 +0000 UTC m=+611.384487294"
Sep 15 21:36:24.564577 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:24.564501    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-xx5vp"
Sep 15 21:36:39.131185 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.131146    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"]
Sep 15 21:36:39.134951 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.134928    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.137437 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.137413    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 21:36:39.138069 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.138031    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-r5v6f\""
Sep 15 21:36:39.138069 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.138034    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 15 21:36:39.138230 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.138071    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 21:36:39.142947 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.142919    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"]
Sep 15 21:36:39.227612 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.227582    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06cebf8b-8c61-4f55-8a83-4b04c172ce40-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.227812 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.227622    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntqts\" (UniqueName: \"kubernetes.io/projected/06cebf8b-8c61-4f55-8a83-4b04c172ce40-kube-api-access-ntqts\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.227812 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.227702    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06cebf8b-8c61-4f55-8a83-4b04c172ce40-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.227812 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.227720    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06cebf8b-8c61-4f55-8a83-4b04c172ce40-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.227812 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.227779    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06cebf8b-8c61-4f55-8a83-4b04c172ce40-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.227983 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.227832    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06cebf8b-8c61-4f55-8a83-4b04c172ce40-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.328972 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.328933    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06cebf8b-8c61-4f55-8a83-4b04c172ce40-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.328972 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.328973    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06cebf8b-8c61-4f55-8a83-4b04c172ce40-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.329214 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.328991    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06cebf8b-8c61-4f55-8a83-4b04c172ce40-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.329214 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.329035    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06cebf8b-8c61-4f55-8a83-4b04c172ce40-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.329214 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.329082    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06cebf8b-8c61-4f55-8a83-4b04c172ce40-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.329214 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.329111    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntqts\" (UniqueName: \"kubernetes.io/projected/06cebf8b-8c61-4f55-8a83-4b04c172ce40-kube-api-access-ntqts\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.329419 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.329398    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06cebf8b-8c61-4f55-8a83-4b04c172ce40-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.329494 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.329439    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06cebf8b-8c61-4f55-8a83-4b04c172ce40-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.329494 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.329469    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06cebf8b-8c61-4f55-8a83-4b04c172ce40-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.331374 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.331355    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06cebf8b-8c61-4f55-8a83-4b04c172ce40-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.331627 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.331610    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06cebf8b-8c61-4f55-8a83-4b04c172ce40-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.337019 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.337002    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntqts\" (UniqueName: \"kubernetes.io/projected/06cebf8b-8c61-4f55-8a83-4b04c172ce40-kube-api-access-ntqts\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q\" (UID: \"06cebf8b-8c61-4f55-8a83-4b04c172ce40\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.441953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.441874    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:36:39.447395 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.447371    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:39.719239 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.719214    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"]
Sep 15 21:36:39.721371 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:36:39.721339    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06cebf8b_8c61_4f55_8a83_4b04c172ce40.slice/crio-503e103941415ece03b29c83ad0607a5e553889889d7b385444170f40c67bbbe WatchSource:0}: Error finding container 503e103941415ece03b29c83ad0607a5e553889889d7b385444170f40c67bbbe: Status 404 returned error can't find the container with id 503e103941415ece03b29c83ad0607a5e553889889d7b385444170f40c67bbbe
Sep 15 21:36:39.722734 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:39.722716    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:36:40.681818 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:40.681785    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q" event={"ID":"06cebf8b-8c61-4f55-8a83-4b04c172ce40","Type":"ContainerStarted","Data":"503e103941415ece03b29c83ad0607a5e553889889d7b385444170f40c67bbbe"}
Sep 15 21:36:41.687970 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:41.687934    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q" event={"ID":"06cebf8b-8c61-4f55-8a83-4b04c172ce40","Type":"ContainerStarted","Data":"7da89482fc9baf7b30e58ae003d1a24d2baffd3f9dcb7e5c4f7e3437f23232cc"}
Sep 15 21:36:41.705083 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:41.705033    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q" podStartSLOduration=1.565556248 podStartE2EDuration="2.705016216s" totalImagesPullingTime="1.139459968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:41.70305362 +0000 UTC m=+639.515062063" watchObservedRunningTime="2026-09-15 21:36:41.705016216 +0000 UTC m=+639.517024659"
Sep 15 21:36:49.447539 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:49.447501    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:49.448040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:49.447604    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:49.460054 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:49.460032    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:49.738851 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:49.738769    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q"
Sep 15 21:36:52.448367 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:52.448327    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:36:58.336875 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.336838    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"]
Sep 15 21:36:58.340418 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.340394    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.342682 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.342652    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 15 21:36:58.351667 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.351634    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"]
Sep 15 21:36:58.406595 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.406539    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a353341f-6fe2-47ed-852e-4e2cb39de18b-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.406830 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.406652    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svgbj\" (UniqueName: \"kubernetes.io/projected/a353341f-6fe2-47ed-852e-4e2cb39de18b-kube-api-access-svgbj\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.406830 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.406716    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a353341f-6fe2-47ed-852e-4e2cb39de18b-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.406830 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.406740    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a353341f-6fe2-47ed-852e-4e2cb39de18b-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.406830 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.406801    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a353341f-6fe2-47ed-852e-4e2cb39de18b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.407045 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.406938    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a353341f-6fe2-47ed-852e-4e2cb39de18b-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.509178 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.509141    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a353341f-6fe2-47ed-852e-4e2cb39de18b-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.509178 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.509178    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a353341f-6fe2-47ed-852e-4e2cb39de18b-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.509460 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.509198    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a353341f-6fe2-47ed-852e-4e2cb39de18b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.509460 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.509255    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a353341f-6fe2-47ed-852e-4e2cb39de18b-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.509460 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.509314    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a353341f-6fe2-47ed-852e-4e2cb39de18b-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.509460 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.509374    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svgbj\" (UniqueName: \"kubernetes.io/projected/a353341f-6fe2-47ed-852e-4e2cb39de18b-kube-api-access-svgbj\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.509850 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.509694    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a353341f-6fe2-47ed-852e-4e2cb39de18b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.509850 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.509808    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a353341f-6fe2-47ed-852e-4e2cb39de18b-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.510048 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.510014    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a353341f-6fe2-47ed-852e-4e2cb39de18b-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.512208 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.512174    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a353341f-6fe2-47ed-852e-4e2cb39de18b-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.512356 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.512178    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a353341f-6fe2-47ed-852e-4e2cb39de18b-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.518433 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.518403    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svgbj\" (UniqueName: \"kubernetes.io/projected/a353341f-6fe2-47ed-852e-4e2cb39de18b-kube-api-access-svgbj\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr\" (UID: \"a353341f-6fe2-47ed-852e-4e2cb39de18b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.651902 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.651853    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:36:58.931367 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:58.931339    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"]
Sep 15 21:36:58.934053 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:36:58.934022    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda353341f_6fe2_47ed_852e_4e2cb39de18b.slice/crio-7a3e5cb5cf78d9f285b6ecdcbd5c43e6ab65b70e8e0f856abb56c26e3ddfd3da WatchSource:0}: Error finding container 7a3e5cb5cf78d9f285b6ecdcbd5c43e6ab65b70e8e0f856abb56c26e3ddfd3da: Status 404 returned error can't find the container with id 7a3e5cb5cf78d9f285b6ecdcbd5c43e6ab65b70e8e0f856abb56c26e3ddfd3da
Sep 15 21:36:59.552045 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:59.551937    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:36:59.775151 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:59.775115    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr" event={"ID":"a353341f-6fe2-47ed-852e-4e2cb39de18b","Type":"ContainerStarted","Data":"ed492704e766ac36d9e234867a6b7369c2b5fa0dc54c58b97fe7130ccfc41134"}
Sep 15 21:36:59.775331 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:59.775171    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr" event={"ID":"a353341f-6fe2-47ed-852e-4e2cb39de18b","Type":"ContainerStarted","Data":"7a3e5cb5cf78d9f285b6ecdcbd5c43e6ab65b70e8e0f856abb56c26e3ddfd3da"}
Sep 15 21:36:59.797060 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:36:59.797004    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr" podStartSLOduration=1.576820764 podStartE2EDuration="1.796983981s" totalImagesPullingTime="220.163217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:36:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:36:59.795313667 +0000 UTC m=+657.607322112" watchObservedRunningTime="2026-09-15 21:36:59.796983981 +0000 UTC m=+657.608992482"
Sep 15 21:37:04.035991 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:04.035954    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:37:08.652477 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:08.652434    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:37:08.652912 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:08.652506    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:37:08.665130 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:08.665096    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:37:08.839096 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:08.839059    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr"
Sep 15 21:37:17.736809 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:17.736750    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:37:36.530352 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:36.530299    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:37:47.441549 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.441496    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"]
Sep 15 21:37:47.445596 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.445561    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.447897 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.447865    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 15 21:37:47.452947 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.452913    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"]
Sep 15 21:37:47.585627 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.585584    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98508da6-aab2-4506-92e2-7a9f7e3467a5-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.585836 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.585659    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98508da6-aab2-4506-92e2-7a9f7e3467a5-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.585836 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.585689    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98508da6-aab2-4506-92e2-7a9f7e3467a5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.585836 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.585708    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98508da6-aab2-4506-92e2-7a9f7e3467a5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.585947 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.585904    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98508da6-aab2-4506-92e2-7a9f7e3467a5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.585947 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.585935    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jntx\" (UniqueName: \"kubernetes.io/projected/98508da6-aab2-4506-92e2-7a9f7e3467a5-kube-api-access-7jntx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.686622 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.686582    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98508da6-aab2-4506-92e2-7a9f7e3467a5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.686622 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.686621    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jntx\" (UniqueName: \"kubernetes.io/projected/98508da6-aab2-4506-92e2-7a9f7e3467a5-kube-api-access-7jntx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.686944 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.686643    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98508da6-aab2-4506-92e2-7a9f7e3467a5-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.686944 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.686677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98508da6-aab2-4506-92e2-7a9f7e3467a5-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.686944 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.686693    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98508da6-aab2-4506-92e2-7a9f7e3467a5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.686944 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.686710    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98508da6-aab2-4506-92e2-7a9f7e3467a5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.687161 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.687080    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98508da6-aab2-4506-92e2-7a9f7e3467a5-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.687217 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.687187    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98508da6-aab2-4506-92e2-7a9f7e3467a5-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.687270 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.687221    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98508da6-aab2-4506-92e2-7a9f7e3467a5-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.689402 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.689374    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98508da6-aab2-4506-92e2-7a9f7e3467a5-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.689543 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.689526    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98508da6-aab2-4506-92e2-7a9f7e3467a5-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.694847 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.694747    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jntx\" (UniqueName: \"kubernetes.io/projected/98508da6-aab2-4506-92e2-7a9f7e3467a5-kube-api-access-7jntx\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-9flvv\" (UID: \"98508da6-aab2-4506-92e2-7a9f7e3467a5\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:47.757650 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:47.757605    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:48.037234 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:48.037206    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"]
Sep 15 21:37:48.039506 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:37:48.039472    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98508da6_aab2_4506_92e2_7a9f7e3467a5.slice/crio-3b6438e96f70941011cecfe4e02e71ad3a2a66e35a08e19c1623261b4db12d27 WatchSource:0}: Error finding container 3b6438e96f70941011cecfe4e02e71ad3a2a66e35a08e19c1623261b4db12d27: Status 404 returned error can't find the container with id 3b6438e96f70941011cecfe4e02e71ad3a2a66e35a08e19c1623261b4db12d27
Sep 15 21:37:49.031737 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:49.031700    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv" event={"ID":"98508da6-aab2-4506-92e2-7a9f7e3467a5","Type":"ContainerStarted","Data":"7b58e8e026075dc7a5bf1b37db371d2bd6572598712a5580440f4b219c0fcc04"}
Sep 15 21:37:49.031737 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:49.031746    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv" event={"ID":"98508da6-aab2-4506-92e2-7a9f7e3467a5","Type":"ContainerStarted","Data":"3b6438e96f70941011cecfe4e02e71ad3a2a66e35a08e19c1623261b4db12d27"}
Sep 15 21:37:49.049610 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:49.049542    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv" podStartSLOduration=1.870643547 podStartE2EDuration="2.049518185s" totalImagesPullingTime="178.874638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:37:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:37:49.048710678 +0000 UTC m=+706.860719134" watchObservedRunningTime="2026-09-15 21:37:49.049518185 +0000 UTC m=+706.861526630"
Sep 15 21:37:49.836249 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:49.836207    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:37:57.758243 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:57.758133    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:57.758243 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:57.758202    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:57.771231 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:57.771201    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:37:58.088409 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:37:58.088321    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-9flvv"
Sep 15 21:39:32.214495 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:32.214415    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7737-307c10"]
Sep 15 21:39:32.217818 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:32.217800    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-307c10"
Sep 15 21:39:32.219936 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:32.219912    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2lhf2\""
Sep 15 21:39:32.220681 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:32.220656    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7737-307c10"]
Sep 15 21:39:32.350582 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:32.350547    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wcdt\" (UniqueName: \"kubernetes.io/projected/34967b0f-bd5d-4ea8-ad40-c9b084f5625d-kube-api-access-4wcdt\") pod \"test-curl-7737-307c10\" (UID: \"34967b0f-bd5d-4ea8-ad40-c9b084f5625d\") " pod="opendatahub/test-curl-7737-307c10"
Sep 15 21:39:32.451559 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:32.451519    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wcdt\" (UniqueName: \"kubernetes.io/projected/34967b0f-bd5d-4ea8-ad40-c9b084f5625d-kube-api-access-4wcdt\") pod \"test-curl-7737-307c10\" (UID: \"34967b0f-bd5d-4ea8-ad40-c9b084f5625d\") " pod="opendatahub/test-curl-7737-307c10"
Sep 15 21:39:32.459152 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:32.459119    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wcdt\" (UniqueName: \"kubernetes.io/projected/34967b0f-bd5d-4ea8-ad40-c9b084f5625d-kube-api-access-4wcdt\") pod \"test-curl-7737-307c10\" (UID: \"34967b0f-bd5d-4ea8-ad40-c9b084f5625d\") " pod="opendatahub/test-curl-7737-307c10"
Sep 15 21:39:32.528136 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:32.528062    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-307c10"
Sep 15 21:39:32.796861 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:32.796770    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7737-307c10"]
Sep 15 21:39:32.799250 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:39:32.799194    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34967b0f_bd5d_4ea8_ad40_c9b084f5625d.slice/crio-77a59820507182606530771f5f1d287beb565d53def24c9dd7ad052a01f66204 WatchSource:0}: Error finding container 77a59820507182606530771f5f1d287beb565d53def24c9dd7ad052a01f66204: Status 404 returned error can't find the container with id 77a59820507182606530771f5f1d287beb565d53def24c9dd7ad052a01f66204
Sep 15 21:39:33.527369 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:33.527327    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7737-307c10" event={"ID":"34967b0f-bd5d-4ea8-ad40-c9b084f5625d","Type":"ContainerStarted","Data":"77a59820507182606530771f5f1d287beb565d53def24c9dd7ad052a01f66204"}
Sep 15 21:39:34.029158 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:34.029117    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:39:36.184707 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:36.184667    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"]
Sep 15 21:39:36.188306 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:36.188283    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"
Sep 15 21:39:36.190639 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:36.190620    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1af1db\""
Sep 15 21:39:36.190780 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:36.190659    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1af1db-dockercfg-6dsjt\""
Sep 15 21:39:36.204916 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:36.204895    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"]
Sep 15 21:39:36.288636 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:36.288607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8299e83-f37c-4ba8-8e14-f5f67691e191-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl\" (UID: \"c8299e83-f37c-4ba8-8e14-f5f67691e191\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"
Sep 15 21:39:36.288797 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:36.288666    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmlbn\" (UniqueName: \"kubernetes.io/projected/c8299e83-f37c-4ba8-8e14-f5f67691e191-kube-api-access-jmlbn\") pod \"payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl\" (UID: \"c8299e83-f37c-4ba8-8e14-f5f67691e191\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"
Sep 15 21:39:36.389873 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:36.389832    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8299e83-f37c-4ba8-8e14-f5f67691e191-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl\" (UID: \"c8299e83-f37c-4ba8-8e14-f5f67691e191\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"
Sep 15 21:39:36.390062 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:36.389913    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmlbn\" (UniqueName: \"kubernetes.io/projected/c8299e83-f37c-4ba8-8e14-f5f67691e191-kube-api-access-jmlbn\") pod \"payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl\" (UID: \"c8299e83-f37c-4ba8-8e14-f5f67691e191\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"
Sep 15 21:39:36.390521 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:36.390503    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8299e83-f37c-4ba8-8e14-f5f67691e191-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl\" (UID: \"c8299e83-f37c-4ba8-8e14-f5f67691e191\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"
Sep 15 21:39:36.398092 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:36.398065    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmlbn\" (UniqueName: \"kubernetes.io/projected/c8299e83-f37c-4ba8-8e14-f5f67691e191-kube-api-access-jmlbn\") pod \"payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl\" (UID: \"c8299e83-f37c-4ba8-8e14-f5f67691e191\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"
Sep 15 21:39:36.498637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:36.498552    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"
Sep 15 21:39:36.546951 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:36.546909    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7737-307c10" event={"ID":"34967b0f-bd5d-4ea8-ad40-c9b084f5625d","Type":"ContainerStarted","Data":"73351407f79396537d314817f354374dca10caf68de2142f0d16783af3b233df"}
Sep 15 21:39:36.562444 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:36.562397    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7737-307c10" podStartSLOduration=1.162689715 podStartE2EDuration="4.562380667s" totalImagesPullingTime="3.399690952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:36.561953521 +0000 UTC m=+814.373961966" watchObservedRunningTime="2026-09-15 21:39:36.562380667 +0000 UTC m=+814.374389112"
Sep 15 21:39:36.799502 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:36.799480    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"]
Sep 15 21:39:36.801700 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:39:36.801665    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8299e83_f37c_4ba8_8e14_f5f67691e191.slice/crio-b559deb19c9ae7b9f53b4c4555d7317748f70466cd36b0b1e33776c75dc5d97f WatchSource:0}: Error finding container b559deb19c9ae7b9f53b4c4555d7317748f70466cd36b0b1e33776c75dc5d97f: Status 404 returned error can't find the container with id b559deb19c9ae7b9f53b4c4555d7317748f70466cd36b0b1e33776c75dc5d97f
Sep 15 21:39:37.188099 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.188065    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7737-307c10"]
Sep 15 21:39:37.197971 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.197947    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7737-307c10"]
Sep 15 21:39:37.312894 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.312863    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7737-7583d5"]
Sep 15 21:39:37.313304 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.313290    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34967b0f-bd5d-4ea8-ad40-c9b084f5625d" containerName="test-curl-7737-307c10"
Sep 15 21:39:37.313344 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.313306    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="34967b0f-bd5d-4ea8-ad40-c9b084f5625d" containerName="test-curl-7737-307c10"
Sep 15 21:39:37.313385 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.313376    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="34967b0f-bd5d-4ea8-ad40-c9b084f5625d" containerName="test-curl-7737-307c10"
Sep 15 21:39:37.316527 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.316511    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-7583d5"
Sep 15 21:39:37.322603 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.322580    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7737-7583d5"]
Sep 15 21:39:37.330289 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.330269    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"]
Sep 15 21:39:37.333899 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.333882    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"
Sep 15 21:39:37.336043 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.336024    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-360e43\""
Sep 15 21:39:37.336141 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.336047    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-360e43-dockercfg-nzhpz\""
Sep 15 21:39:37.346955 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.346864    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"]
Sep 15 21:39:37.368493 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.368461    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"]
Sep 15 21:39:37.372150 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.372135    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"
Sep 15 21:39:37.380385 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.380363    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"]
Sep 15 21:39:37.397929 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.397907    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv96h\" (UniqueName: \"kubernetes.io/projected/37b09c76-23af-4676-89c4-925e680e957b-kube-api-access-kv96h\") pod \"test-curl-7737-7583d5\" (UID: \"37b09c76-23af-4676-89c4-925e680e957b\") " pod="opendatahub/test-curl-7737-7583d5"
Sep 15 21:39:37.499236 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.499155    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c882ebae-b6ef-4b97-8162-0dc3e3ee03aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p\" (UID: \"c882ebae-b6ef-4b97-8162-0dc3e3ee03aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"
Sep 15 21:39:37.499236 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.499220    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7b0d4c2-d78a-4580-821d-19ed1a49f238-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz\" (UID: \"b7b0d4c2-d78a-4580-821d-19ed1a49f238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"
Sep 15 21:39:37.499430 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.499334    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv96h\" (UniqueName: \"kubernetes.io/projected/37b09c76-23af-4676-89c4-925e680e957b-kube-api-access-kv96h\") pod \"test-curl-7737-7583d5\" (UID: \"37b09c76-23af-4676-89c4-925e680e957b\") " pod="opendatahub/test-curl-7737-7583d5"
Sep 15 21:39:37.499430 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.499389    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gk67\" (UniqueName: \"kubernetes.io/projected/c882ebae-b6ef-4b97-8162-0dc3e3ee03aa-kube-api-access-5gk67\") pod \"payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p\" (UID: \"c882ebae-b6ef-4b97-8162-0dc3e3ee03aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"
Sep 15 21:39:37.499509 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.499447    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbsl5\" (UniqueName: \"kubernetes.io/projected/b7b0d4c2-d78a-4580-821d-19ed1a49f238-kube-api-access-wbsl5\") pod \"payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz\" (UID: \"b7b0d4c2-d78a-4580-821d-19ed1a49f238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"
Sep 15 21:39:37.507096 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.507065    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv96h\" (UniqueName: \"kubernetes.io/projected/37b09c76-23af-4676-89c4-925e680e957b-kube-api-access-kv96h\") pod \"test-curl-7737-7583d5\" (UID: \"37b09c76-23af-4676-89c4-925e680e957b\") " pod="opendatahub/test-curl-7737-7583d5"
Sep 15 21:39:37.553408 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.553373    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl" event={"ID":"c8299e83-f37c-4ba8-8e14-f5f67691e191","Type":"ContainerStarted","Data":"1ae774baff3d9fba698c9b57a13c96e865c59ebcd5d8f3a7994a46ce9d305806"}
Sep 15 21:39:37.553569 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.553428    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl" event={"ID":"c8299e83-f37c-4ba8-8e14-f5f67691e191","Type":"ContainerStarted","Data":"b559deb19c9ae7b9f53b4c4555d7317748f70466cd36b0b1e33776c75dc5d97f"}
Sep 15 21:39:37.553569 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.553469    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"
Sep 15 21:39:37.553685 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:39:37.553609    2575 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7737-307c10" container="test-curl-7737-307c10"
Sep 15 21:39:37.570895 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.570847    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl" podStartSLOduration=1.570832698 podStartE2EDuration="1.570832698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:37.569490146 +0000 UTC m=+815.381498593" watchObservedRunningTime="2026-09-15 21:39:37.570832698 +0000 UTC m=+815.382841141"
Sep 15 21:39:37.600178 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.600156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7b0d4c2-d78a-4580-821d-19ed1a49f238-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz\" (UID: \"b7b0d4c2-d78a-4580-821d-19ed1a49f238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"
Sep 15 21:39:37.600310 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.600212    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gk67\" (UniqueName: \"kubernetes.io/projected/c882ebae-b6ef-4b97-8162-0dc3e3ee03aa-kube-api-access-5gk67\") pod \"payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p\" (UID: \"c882ebae-b6ef-4b97-8162-0dc3e3ee03aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"
Sep 15 21:39:37.600310 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.600241    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbsl5\" (UniqueName: \"kubernetes.io/projected/b7b0d4c2-d78a-4580-821d-19ed1a49f238-kube-api-access-wbsl5\") pod \"payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz\" (UID: \"b7b0d4c2-d78a-4580-821d-19ed1a49f238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"
Sep 15 21:39:37.600310 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.600258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c882ebae-b6ef-4b97-8162-0dc3e3ee03aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p\" (UID: \"c882ebae-b6ef-4b97-8162-0dc3e3ee03aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"
Sep 15 21:39:37.600984 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.600967    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c882ebae-b6ef-4b97-8162-0dc3e3ee03aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p\" (UID: \"c882ebae-b6ef-4b97-8162-0dc3e3ee03aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"
Sep 15 21:39:37.601038 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.601022    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7b0d4c2-d78a-4580-821d-19ed1a49f238-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz\" (UID: \"b7b0d4c2-d78a-4580-821d-19ed1a49f238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"
Sep 15 21:39:37.609547 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.609524    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbsl5\" (UniqueName: \"kubernetes.io/projected/b7b0d4c2-d78a-4580-821d-19ed1a49f238-kube-api-access-wbsl5\") pod \"payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz\" (UID: \"b7b0d4c2-d78a-4580-821d-19ed1a49f238\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"
Sep 15 21:39:37.609649 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.609573    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gk67\" (UniqueName: \"kubernetes.io/projected/c882ebae-b6ef-4b97-8162-0dc3e3ee03aa-kube-api-access-5gk67\") pod \"payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p\" (UID: \"c882ebae-b6ef-4b97-8162-0dc3e3ee03aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"
Sep 15 21:39:37.626462 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.626439    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-7583d5"
Sep 15 21:39:37.651963 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.651936    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"
Sep 15 21:39:37.684381 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.683966    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"
Sep 15 21:39:37.903161 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.903139    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7737-7583d5"]
Sep 15 21:39:37.904860 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:39:37.904834    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37b09c76_23af_4676_89c4_925e680e957b.slice/crio-9a5dd19bae180a617ec771c600b74ba652e736ea3d96b1df34cb994b80a93c8b WatchSource:0}: Error finding container 9a5dd19bae180a617ec771c600b74ba652e736ea3d96b1df34cb994b80a93c8b: Status 404 returned error can't find the container with id 9a5dd19bae180a617ec771c600b74ba652e736ea3d96b1df34cb994b80a93c8b
Sep 15 21:39:37.941294 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.941262    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"]
Sep 15 21:39:37.942173 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:39:37.942144    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc882ebae_b6ef_4b97_8162_0dc3e3ee03aa.slice/crio-170069c64ca790aec2aa503ec248c791b648ebf9fe6090451223dacbadd5d0c7 WatchSource:0}: Error finding container 170069c64ca790aec2aa503ec248c791b648ebf9fe6090451223dacbadd5d0c7: Status 404 returned error can't find the container with id 170069c64ca790aec2aa503ec248c791b648ebf9fe6090451223dacbadd5d0c7
Sep 15 21:39:37.972845 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:37.972821    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"]
Sep 15 21:39:37.976966 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:39:37.976926    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b0d4c2_d78a_4580_821d_19ed1a49f238.slice/crio-a8d2fd8d1d91a6544f98248e271d1a085af0a61b7eb5713169592b21ac759ef3 WatchSource:0}: Error finding container a8d2fd8d1d91a6544f98248e271d1a085af0a61b7eb5713169592b21ac759ef3: Status 404 returned error can't find the container with id a8d2fd8d1d91a6544f98248e271d1a085af0a61b7eb5713169592b21ac759ef3
Sep 15 21:39:38.561032 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:38.560996    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p" event={"ID":"c882ebae-b6ef-4b97-8162-0dc3e3ee03aa","Type":"ContainerStarted","Data":"da6e02d70cb03f8b802143980e660cf5d62808cbcafac94546296f500db29cea"}
Sep 15 21:39:38.561510 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:38.561184    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"
Sep 15 21:39:38.561510 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:38.561222    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p" event={"ID":"c882ebae-b6ef-4b97-8162-0dc3e3ee03aa","Type":"ContainerStarted","Data":"170069c64ca790aec2aa503ec248c791b648ebf9fe6090451223dacbadd5d0c7"}
Sep 15 21:39:38.562651 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:38.562628    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7737-7583d5" event={"ID":"37b09c76-23af-4676-89c4-925e680e957b","Type":"ContainerStarted","Data":"2555c0fb20512eee6c9fe8e086f67184437ed0cb6f59fbf3fa4e9e3cdd02ac8c"}
Sep 15 21:39:38.562741 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:38.562662    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7737-7583d5" event={"ID":"37b09c76-23af-4676-89c4-925e680e957b","Type":"ContainerStarted","Data":"9a5dd19bae180a617ec771c600b74ba652e736ea3d96b1df34cb994b80a93c8b"}
Sep 15 21:39:38.564454 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:38.564417    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz" event={"ID":"b7b0d4c2-d78a-4580-821d-19ed1a49f238","Type":"ContainerStarted","Data":"0619e1207ebef1195049ccadc768b4090114f6bda722c21c35c8031dc3aecfcd"}
Sep 15 21:39:38.564646 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:38.564616    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7737-307c10" podUID="34967b0f-bd5d-4ea8-ad40-c9b084f5625d" containerName="test-curl-7737-307c10" containerID="cri-o://73351407f79396537d314817f354374dca10caf68de2142f0d16783af3b233df" gracePeriod=2
Sep 15 21:39:38.564714 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:38.564658    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"
Sep 15 21:39:38.564714 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:38.564689    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz" event={"ID":"b7b0d4c2-d78a-4580-821d-19ed1a49f238","Type":"ContainerStarted","Data":"a8d2fd8d1d91a6544f98248e271d1a085af0a61b7eb5713169592b21ac759ef3"}
Sep 15 21:39:38.578663 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:38.578621    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p" podStartSLOduration=1.57860574 podStartE2EDuration="1.57860574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:38.577112187 +0000 UTC m=+816.389120632" watchObservedRunningTime="2026-09-15 21:39:38.57860574 +0000 UTC m=+816.390614184"
Sep 15 21:39:38.593453 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:39:38.593418    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7737-307c10\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="34967b0f-bd5d-4ea8-ad40-c9b084f5625d" pod="opendatahub/test-curl-7737-307c10"
Sep 15 21:39:38.593573 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:38.593444    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz" podStartSLOduration=1.593426076 podStartE2EDuration="1.593426076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:38.591548669 +0000 UTC m=+816.403557119" watchObservedRunningTime="2026-09-15 21:39:38.593426076 +0000 UTC m=+816.405434524"
Sep 15 21:39:38.606295 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:38.606251    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7737-7583d5" podStartSLOduration=1.606238928 podStartE2EDuration="1.606238928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:38.605419555 +0000 UTC m=+816.417428010" watchObservedRunningTime="2026-09-15 21:39:38.606238928 +0000 UTC m=+816.418247371"
Sep 15 21:39:39.202972 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.202930    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7737-7583d5"]
Sep 15 21:39:39.205303 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.205279    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7737-7583d5"]
Sep 15 21:39:39.570239 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:39:39.570208    2575 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7737-7583d5" container="test-curl-7737-7583d5"
Sep 15 21:39:39.603164 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.603136    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"]
Sep 15 21:39:39.603560 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.603549    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b09c76-23af-4676-89c4-925e680e957b" containerName="test-curl-7737-7583d5"
Sep 15 21:39:39.603615 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.603563    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b09c76-23af-4676-89c4-925e680e957b" containerName="test-curl-7737-7583d5"
Sep 15 21:39:39.603661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.603643    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b09c76-23af-4676-89c4-925e680e957b" containerName="test-curl-7737-7583d5"
Sep 15 21:39:39.613956 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.613932    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.616409 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.616386    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3ba8d384-openshift-default-dockercfg-2cmck\""
Sep 15 21:39:39.616877 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.616711    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 21:39:39.618260 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.618239    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"]
Sep 15 21:39:39.719246 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.719216    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-istio-envoy\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.719246 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.719247    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-istio-data\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.719511 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.719287    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-credential-socket\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.719511 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.719387    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b8df82-039f-46ea-8800-4f9599f6916a-istiod-ca-cert\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.719511 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.719462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-workload-certs\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.719511 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.719509    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b8df82-039f-46ea-8800-4f9599f6916a-istio-token\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.719712 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.719672    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b8df82-039f-46ea-8800-4f9599f6916a-istio-podinfo\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.719712 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.719706    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-workload-socket\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.719849 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.719833    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8st8f\" (UniqueName: \"kubernetes.io/projected/76b8df82-039f-46ea-8800-4f9599f6916a-kube-api-access-8st8f\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.820794 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.820698    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b8df82-039f-46ea-8800-4f9599f6916a-istio-podinfo\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.820794 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.820735    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-workload-socket\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.821162 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.820810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8st8f\" (UniqueName: \"kubernetes.io/projected/76b8df82-039f-46ea-8800-4f9599f6916a-kube-api-access-8st8f\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.821162 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.820837    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-istio-envoy\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.821162 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.820867    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-istio-data\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.821162 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.820905    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-credential-socket\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.821162 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.820963    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b8df82-039f-46ea-8800-4f9599f6916a-istiod-ca-cert\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.821162 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.821025    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-workload-certs\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.821162 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.821070    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b8df82-039f-46ea-8800-4f9599f6916a-istio-token\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.821527 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.821234    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-istio-data\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.821527 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.821303    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-credential-socket\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.821527 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.821385    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-workload-certs\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.821527 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.821478    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-workload-socket\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.822238 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.822186    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b8df82-039f-46ea-8800-4f9599f6916a-istiod-ca-cert\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.823362 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.823343    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-istio-envoy\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.823717 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.823694    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b8df82-039f-46ea-8800-4f9599f6916a-istio-podinfo\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.829104 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.829083    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b8df82-039f-46ea-8800-4f9599f6916a-istio-token\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.829361 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.829342    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8st8f\" (UniqueName: \"kubernetes.io/projected/76b8df82-039f-46ea-8800-4f9599f6916a-kube-api-access-8st8f\") pod \"e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") " pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:39.927176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:39.927142    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:40.210131 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:40.210104    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"]
Sep 15 21:39:40.214057 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:39:40.214022    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b8df82_039f_46ea_8800_4f9599f6916a.slice/crio-9822c88ab11d4b7ecb3658ea7ad0abba6cf2f4bc783a11f39d8cee6543f712f4 WatchSource:0}: Error finding container 9822c88ab11d4b7ecb3658ea7ad0abba6cf2f4bc783a11f39d8cee6543f712f4: Status 404 returned error can't find the container with id 9822c88ab11d4b7ecb3658ea7ad0abba6cf2f4bc783a11f39d8cee6543f712f4
Sep 15 21:39:40.578162 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:40.577948    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7" event={"ID":"76b8df82-039f-46ea-8800-4f9599f6916a","Type":"ContainerStarted","Data":"9822c88ab11d4b7ecb3658ea7ad0abba6cf2f4bc783a11f39d8cee6543f712f4"}
Sep 15 21:39:40.578162 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:40.578023    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7737-7583d5" podUID="37b09c76-23af-4676-89c4-925e680e957b" containerName="test-curl-7737-7583d5" containerID="cri-o://2555c0fb20512eee6c9fe8e086f67184437ed0cb6f59fbf3fa4e9e3cdd02ac8c" gracePeriod=2
Sep 15 21:39:40.579767 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:39:40.579731    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7737-7583d5\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="37b09c76-23af-4676-89c4-925e680e957b" pod="opendatahub/test-curl-7737-7583d5"
Sep 15 21:39:40.733906 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:40.733884    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-307c10"
Sep 15 21:39:40.736000 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:39:40.735979    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7737-7583d5\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="37b09c76-23af-4676-89c4-925e680e957b" pod="opendatahub/test-curl-7737-7583d5"
Sep 15 21:39:40.829791 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:40.829383    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wcdt\" (UniqueName: \"kubernetes.io/projected/34967b0f-bd5d-4ea8-ad40-c9b084f5625d-kube-api-access-4wcdt\") pod \"34967b0f-bd5d-4ea8-ad40-c9b084f5625d\" (UID: \"34967b0f-bd5d-4ea8-ad40-c9b084f5625d\") "
Sep 15 21:39:40.832282 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:40.832248    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34967b0f-bd5d-4ea8-ad40-c9b084f5625d-kube-api-access-4wcdt" (OuterVolumeSpecName: "kube-api-access-4wcdt") pod "34967b0f-bd5d-4ea8-ad40-c9b084f5625d" (UID: "34967b0f-bd5d-4ea8-ad40-c9b084f5625d"). InnerVolumeSpecName "kube-api-access-4wcdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:39:40.930600 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:40.930573    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wcdt\" (UniqueName: \"kubernetes.io/projected/34967b0f-bd5d-4ea8-ad40-c9b084f5625d-kube-api-access-4wcdt\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:39:41.586054 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:41.586018    2575 generic.go:356] "Generic (PLEG): container finished" podID="34967b0f-bd5d-4ea8-ad40-c9b084f5625d" containerID="73351407f79396537d314817f354374dca10caf68de2142f0d16783af3b233df" exitCode=137
Sep 15 21:39:41.586528 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:41.586083    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-307c10"
Sep 15 21:39:41.586528 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:41.586139    2575 scope.go:117] "RemoveContainer" containerID="73351407f79396537d314817f354374dca10caf68de2142f0d16783af3b233df"
Sep 15 21:39:41.595915 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:41.595873    2575 scope.go:117] "RemoveContainer" containerID="73351407f79396537d314817f354374dca10caf68de2142f0d16783af3b233df"
Sep 15 21:39:41.596256 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:39:41.596221    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73351407f79396537d314817f354374dca10caf68de2142f0d16783af3b233df\": container with ID starting with 73351407f79396537d314817f354374dca10caf68de2142f0d16783af3b233df not found: ID does not exist" containerID="73351407f79396537d314817f354374dca10caf68de2142f0d16783af3b233df"
Sep 15 21:39:41.596470 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:41.596266    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73351407f79396537d314817f354374dca10caf68de2142f0d16783af3b233df"} err="failed to get container status \"73351407f79396537d314817f354374dca10caf68de2142f0d16783af3b233df\": rpc error: code = NotFound desc = could not find container \"73351407f79396537d314817f354374dca10caf68de2142f0d16783af3b233df\": container with ID starting with 73351407f79396537d314817f354374dca10caf68de2142f0d16783af3b233df not found: ID does not exist"
Sep 15 21:39:42.748984 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:42.748948    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34967b0f-bd5d-4ea8-ad40-c9b084f5625d" path="/var/lib/kubelet/pods/34967b0f-bd5d-4ea8-ad40-c9b084f5625d/volumes"
Sep 15 21:39:42.864650 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:42.864606    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:39:42.864740 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:42.864683    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:39:42.864740 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:42.864711    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:39:42.928195 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:42.928174    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-7583d5"
Sep 15 21:39:42.930023 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:39:42.930000    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7737-7583d5\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="37b09c76-23af-4676-89c4-925e680e957b" pod="opendatahub/test-curl-7737-7583d5"
Sep 15 21:39:43.051259 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:43.051231    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kv96h\" (UniqueName: \"kubernetes.io/projected/37b09c76-23af-4676-89c4-925e680e957b-kube-api-access-kv96h\") pod \"37b09c76-23af-4676-89c4-925e680e957b\" (UID: \"37b09c76-23af-4676-89c4-925e680e957b\") "
Sep 15 21:39:43.053588 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:43.053561    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b09c76-23af-4676-89c4-925e680e957b-kube-api-access-kv96h" (OuterVolumeSpecName: "kube-api-access-kv96h") pod "37b09c76-23af-4676-89c4-925e680e957b" (UID: "37b09c76-23af-4676-89c4-925e680e957b"). InnerVolumeSpecName "kube-api-access-kv96h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:39:43.153028 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:43.152957    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv96h\" (UniqueName: \"kubernetes.io/projected/37b09c76-23af-4676-89c4-925e680e957b-kube-api-access-kv96h\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:39:43.599941 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:43.599910    2575 generic.go:356] "Generic (PLEG): container finished" podID="37b09c76-23af-4676-89c4-925e680e957b" containerID="2555c0fb20512eee6c9fe8e086f67184437ed0cb6f59fbf3fa4e9e3cdd02ac8c" exitCode=137
Sep 15 21:39:43.600138 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:43.599962    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7737-7583d5"
Sep 15 21:39:43.600138 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:43.600006    2575 scope.go:117] "RemoveContainer" containerID="2555c0fb20512eee6c9fe8e086f67184437ed0cb6f59fbf3fa4e9e3cdd02ac8c"
Sep 15 21:39:43.601555 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:43.601527    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7" event={"ID":"76b8df82-039f-46ea-8800-4f9599f6916a","Type":"ContainerStarted","Data":"cd7f45acbcac78b69230c34cb9de40aa983c7b42cbc4d592a230b653fc41bd4f"}
Sep 15 21:39:43.601905 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:39:43.601880    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7737-7583d5\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="37b09c76-23af-4676-89c4-925e680e957b" pod="opendatahub/test-curl-7737-7583d5"
Sep 15 21:39:43.609372 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:43.609353    2575 scope.go:117] "RemoveContainer" containerID="2555c0fb20512eee6c9fe8e086f67184437ed0cb6f59fbf3fa4e9e3cdd02ac8c"
Sep 15 21:39:43.609661 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:39:43.609644    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2555c0fb20512eee6c9fe8e086f67184437ed0cb6f59fbf3fa4e9e3cdd02ac8c\": container with ID starting with 2555c0fb20512eee6c9fe8e086f67184437ed0cb6f59fbf3fa4e9e3cdd02ac8c not found: ID does not exist" containerID="2555c0fb20512eee6c9fe8e086f67184437ed0cb6f59fbf3fa4e9e3cdd02ac8c"
Sep 15 21:39:43.609725 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:43.609669    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2555c0fb20512eee6c9fe8e086f67184437ed0cb6f59fbf3fa4e9e3cdd02ac8c"} err="failed to get container status \"2555c0fb20512eee6c9fe8e086f67184437ed0cb6f59fbf3fa4e9e3cdd02ac8c\": rpc error: code = NotFound desc = could not find container \"2555c0fb20512eee6c9fe8e086f67184437ed0cb6f59fbf3fa4e9e3cdd02ac8c\": container with ID starting with 2555c0fb20512eee6c9fe8e086f67184437ed0cb6f59fbf3fa4e9e3cdd02ac8c not found: ID does not exist"
Sep 15 21:39:43.621886 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:43.621840    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7" podStartSLOduration=1.973645724 podStartE2EDuration="4.621822837s" totalImagesPullingTime="2.648177113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:43.621087222 +0000 UTC m=+821.433095670" watchObservedRunningTime="2026-09-15 21:39:43.621822837 +0000 UTC m=+821.433831283"
Sep 15 21:39:43.622864 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:39:43.622831    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7737-7583d5\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="37b09c76-23af-4676-89c4-925e680e957b" pod="opendatahub/test-curl-7737-7583d5"
Sep 15 21:39:43.624325 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:39:43.624304    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7737-7583d5\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="37b09c76-23af-4676-89c4-925e680e957b" pod="opendatahub/test-curl-7737-7583d5"
Sep 15 21:39:43.927706 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:43.927624    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:43.932714 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:43.932690    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:43.950814 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:39:43.950781    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7737-7583d5\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="37b09c76-23af-4676-89c4-925e680e957b" pod="opendatahub/test-curl-7737-7583d5"
Sep 15 21:39:44.607849 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:44.607813    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:44.609135 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:44.609076    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:39:44.610944 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:39:44.610920    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7737-7583d5\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="37b09c76-23af-4676-89c4-925e680e957b" pod="opendatahub/test-curl-7737-7583d5"
Sep 15 21:39:44.747610 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:44.747575    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b09c76-23af-4676-89c4-925e680e957b" path="/var/lib/kubelet/pods/37b09c76-23af-4676-89c4-925e680e957b/volumes"
Sep 15 21:39:46.621331 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.621298    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"]
Sep 15 21:39:46.626152 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.626136    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"
Sep 15 21:39:46.628645 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.628623    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-5416b7\""
Sep 15 21:39:46.628743 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.628642    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-5416b7-dockercfg-bw6v7\""
Sep 15 21:39:46.639646 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.639625    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"]
Sep 15 21:39:46.658003 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.657967    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"]
Sep 15 21:39:46.661550 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.661535    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"
Sep 15 21:39:46.669890 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.669868    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"]
Sep 15 21:39:46.786645 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.786609    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71381e8f-030b-4482-a8fc-1a1133365237-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw\" (UID: \"71381e8f-030b-4482-a8fc-1a1133365237\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"
Sep 15 21:39:46.786888 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.786747    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7880acb1-c1d9-4509-ae56-1cd81a2ca6c8-plugins-config-volume\") pod \"payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr\" (UID: \"7880acb1-c1d9-4509-ae56-1cd81a2ca6c8\") " pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"
Sep 15 21:39:46.786888 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.786829    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mvc9\" (UniqueName: \"kubernetes.io/projected/7880acb1-c1d9-4509-ae56-1cd81a2ca6c8-kube-api-access-7mvc9\") pod \"payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr\" (UID: \"7880acb1-c1d9-4509-ae56-1cd81a2ca6c8\") " pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"
Sep 15 21:39:46.787016 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.786885    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4rdk\" (UniqueName: \"kubernetes.io/projected/71381e8f-030b-4482-a8fc-1a1133365237-kube-api-access-h4rdk\") pod \"payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw\" (UID: \"71381e8f-030b-4482-a8fc-1a1133365237\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"
Sep 15 21:39:46.887726 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.887618    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71381e8f-030b-4482-a8fc-1a1133365237-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw\" (UID: \"71381e8f-030b-4482-a8fc-1a1133365237\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"
Sep 15 21:39:46.887726 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.887681    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7880acb1-c1d9-4509-ae56-1cd81a2ca6c8-plugins-config-volume\") pod \"payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr\" (UID: \"7880acb1-c1d9-4509-ae56-1cd81a2ca6c8\") " pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"
Sep 15 21:39:46.887726 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.887717    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mvc9\" (UniqueName: \"kubernetes.io/projected/7880acb1-c1d9-4509-ae56-1cd81a2ca6c8-kube-api-access-7mvc9\") pod \"payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr\" (UID: \"7880acb1-c1d9-4509-ae56-1cd81a2ca6c8\") " pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"
Sep 15 21:39:46.888037 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.887828    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4rdk\" (UniqueName: \"kubernetes.io/projected/71381e8f-030b-4482-a8fc-1a1133365237-kube-api-access-h4rdk\") pod \"payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw\" (UID: \"71381e8f-030b-4482-a8fc-1a1133365237\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"
Sep 15 21:39:46.888366 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.888348    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71381e8f-030b-4482-a8fc-1a1133365237-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw\" (UID: \"71381e8f-030b-4482-a8fc-1a1133365237\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"
Sep 15 21:39:46.888420 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.888406    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7880acb1-c1d9-4509-ae56-1cd81a2ca6c8-plugins-config-volume\") pod \"payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr\" (UID: \"7880acb1-c1d9-4509-ae56-1cd81a2ca6c8\") " pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"
Sep 15 21:39:46.900659 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.900632    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4rdk\" (UniqueName: \"kubernetes.io/projected/71381e8f-030b-4482-a8fc-1a1133365237-kube-api-access-h4rdk\") pod \"payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw\" (UID: \"71381e8f-030b-4482-a8fc-1a1133365237\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"
Sep 15 21:39:46.901186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.901167    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mvc9\" (UniqueName: \"kubernetes.io/projected/7880acb1-c1d9-4509-ae56-1cd81a2ca6c8-kube-api-access-7mvc9\") pod \"payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr\" (UID: \"7880acb1-c1d9-4509-ae56-1cd81a2ca6c8\") " pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"
Sep 15 21:39:46.936328 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.936301    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"
Sep 15 21:39:46.972620 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:46.972593    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"
Sep 15 21:39:47.213252 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.213181    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"]
Sep 15 21:39:47.216981 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:39:47.216955    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71381e8f_030b_4482_a8fc_1a1133365237.slice/crio-06a1145e74e0021cf180cfc75ae8513d86d1cf9a4124f1dfd698f2ed33371dd6 WatchSource:0}: Error finding container 06a1145e74e0021cf180cfc75ae8513d86d1cf9a4124f1dfd698f2ed33371dd6: Status 404 returned error can't find the container with id 06a1145e74e0021cf180cfc75ae8513d86d1cf9a4124f1dfd698f2ed33371dd6
Sep 15 21:39:47.258038 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.258011    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"]
Sep 15 21:39:47.260818 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:39:47.260787    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7880acb1_c1d9_4509_ae56_1cd81a2ca6c8.slice/crio-d7235fb7f004abff176e24b9237b195c09767d61ab5eb20029b451d53b8775ae WatchSource:0}: Error finding container d7235fb7f004abff176e24b9237b195c09767d61ab5eb20029b451d53b8775ae: Status 404 returned error can't find the container with id d7235fb7f004abff176e24b9237b195c09767d61ab5eb20029b451d53b8775ae
Sep 15 21:39:47.528444 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.528365    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"]
Sep 15 21:39:47.531944 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.531927    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"
Sep 15 21:39:47.534274 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.534245    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3ba8d384-dockercfg-x9vf7\""
Sep 15 21:39:47.534403 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.534292    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3ba8d384\""
Sep 15 21:39:47.541912 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.541885    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"]
Sep 15 21:39:47.549654 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.549625    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"]
Sep 15 21:39:47.554063 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.554042    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"
Sep 15 21:39:47.564984 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.564958    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"]
Sep 15 21:39:47.622975 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.622939    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr" event={"ID":"7880acb1-c1d9-4509-ae56-1cd81a2ca6c8","Type":"ContainerStarted","Data":"773cac1d44fa13f24e02ab7df85e5b924902de47c6c15f565ddab7f58a7ac557"}
Sep 15 21:39:47.623412 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.623024    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr" event={"ID":"7880acb1-c1d9-4509-ae56-1cd81a2ca6c8","Type":"ContainerStarted","Data":"d7235fb7f004abff176e24b9237b195c09767d61ab5eb20029b451d53b8775ae"}
Sep 15 21:39:47.623412 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.623064    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"
Sep 15 21:39:47.624626 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.624602    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw" event={"ID":"71381e8f-030b-4482-a8fc-1a1133365237","Type":"ContainerStarted","Data":"c9a830540fdeb5804c2c80915e2669fa982fb599d64ca3fe3610543146fe371a"}
Sep 15 21:39:47.624749 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.624639    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw" event={"ID":"71381e8f-030b-4482-a8fc-1a1133365237","Type":"ContainerStarted","Data":"06a1145e74e0021cf180cfc75ae8513d86d1cf9a4124f1dfd698f2ed33371dd6"}
Sep 15 21:39:47.624749 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.624660    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"
Sep 15 21:39:47.659538 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.659477    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw" podStartSLOduration=1.659456405 podStartE2EDuration="1.659456405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:47.657908202 +0000 UTC m=+825.469916651" watchObservedRunningTime="2026-09-15 21:39:47.659456405 +0000 UTC m=+825.471464851"
Sep 15 21:39:47.663075 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.662775    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr" podStartSLOduration=1.662735181 podStartE2EDuration="1.662735181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:47.639465999 +0000 UTC m=+825.451474461" watchObservedRunningTime="2026-09-15 21:39:47.662735181 +0000 UTC m=+825.474743629"
Sep 15 21:39:47.694292 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.694252    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e087cc-6e58-42c6-9905-c1a045a35399-plugins-config-volume\") pod \"payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4\" (UID: \"45e087cc-6e58-42c6-9905-c1a045a35399\") " pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"
Sep 15 21:39:47.694427 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.694312    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767add36-0585-4d1f-956f-45b6cf6d6e7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6\" (UID: \"767add36-0585-4d1f-956f-45b6cf6d6e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"
Sep 15 21:39:47.694427 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.694353    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzgfs\" (UniqueName: \"kubernetes.io/projected/45e087cc-6e58-42c6-9905-c1a045a35399-kube-api-access-wzgfs\") pod \"payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4\" (UID: \"45e087cc-6e58-42c6-9905-c1a045a35399\") " pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"
Sep 15 21:39:47.694427 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.694387    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qklfn\" (UniqueName: \"kubernetes.io/projected/767add36-0585-4d1f-956f-45b6cf6d6e7a-kube-api-access-qklfn\") pod \"payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6\" (UID: \"767add36-0585-4d1f-956f-45b6cf6d6e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"
Sep 15 21:39:47.795342 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.795261    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e087cc-6e58-42c6-9905-c1a045a35399-plugins-config-volume\") pod \"payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4\" (UID: \"45e087cc-6e58-42c6-9905-c1a045a35399\") " pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"
Sep 15 21:39:47.795342 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.795332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767add36-0585-4d1f-956f-45b6cf6d6e7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6\" (UID: \"767add36-0585-4d1f-956f-45b6cf6d6e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"
Sep 15 21:39:47.795566 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.795388    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzgfs\" (UniqueName: \"kubernetes.io/projected/45e087cc-6e58-42c6-9905-c1a045a35399-kube-api-access-wzgfs\") pod \"payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4\" (UID: \"45e087cc-6e58-42c6-9905-c1a045a35399\") " pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"
Sep 15 21:39:47.795566 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.795427    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qklfn\" (UniqueName: \"kubernetes.io/projected/767add36-0585-4d1f-956f-45b6cf6d6e7a-kube-api-access-qklfn\") pod \"payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6\" (UID: \"767add36-0585-4d1f-956f-45b6cf6d6e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"
Sep 15 21:39:47.796069 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.796043    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e087cc-6e58-42c6-9905-c1a045a35399-plugins-config-volume\") pod \"payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4\" (UID: \"45e087cc-6e58-42c6-9905-c1a045a35399\") " pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"
Sep 15 21:39:47.796156 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.796140    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767add36-0585-4d1f-956f-45b6cf6d6e7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6\" (UID: \"767add36-0585-4d1f-956f-45b6cf6d6e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"
Sep 15 21:39:47.804431 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.804408    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzgfs\" (UniqueName: \"kubernetes.io/projected/45e087cc-6e58-42c6-9905-c1a045a35399-kube-api-access-wzgfs\") pod \"payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4\" (UID: \"45e087cc-6e58-42c6-9905-c1a045a35399\") " pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"
Sep 15 21:39:47.804555 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.804452    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qklfn\" (UniqueName: \"kubernetes.io/projected/767add36-0585-4d1f-956f-45b6cf6d6e7a-kube-api-access-qklfn\") pod \"payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6\" (UID: \"767add36-0585-4d1f-956f-45b6cf6d6e7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"
Sep 15 21:39:47.844625 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.844599    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"
Sep 15 21:39:47.866820 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:47.866793    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"
Sep 15 21:39:48.121929 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:48.121903    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"]
Sep 15 21:39:48.123381 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:39:48.123357    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod767add36_0585_4d1f_956f_45b6cf6d6e7a.slice/crio-2cb0b0c1f747d9d491d854f0a487cd72edfcc16f03223ee7553ffe3f64d6c55a WatchSource:0}: Error finding container 2cb0b0c1f747d9d491d854f0a487cd72edfcc16f03223ee7553ffe3f64d6c55a: Status 404 returned error can't find the container with id 2cb0b0c1f747d9d491d854f0a487cd72edfcc16f03223ee7553ffe3f64d6c55a
Sep 15 21:39:48.151681 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:48.151660    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"]
Sep 15 21:39:48.157971 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:39:48.157945    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e087cc_6e58_42c6_9905_c1a045a35399.slice/crio-85338f894b0041af9cb6cbe6201259e828edcad0dae0c60cbedf4b25f871f4cb WatchSource:0}: Error finding container 85338f894b0041af9cb6cbe6201259e828edcad0dae0c60cbedf4b25f871f4cb: Status 404 returned error can't find the container with id 85338f894b0041af9cb6cbe6201259e828edcad0dae0c60cbedf4b25f871f4cb
Sep 15 21:39:48.567057 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:48.566968    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"
Sep 15 21:39:48.631840 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:48.631800    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6" event={"ID":"767add36-0585-4d1f-956f-45b6cf6d6e7a","Type":"ContainerStarted","Data":"be59937f44ae47b227a150df0f19b351a5069df58154843232a52f1b29c80e29"}
Sep 15 21:39:48.632285 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:48.631863    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6" event={"ID":"767add36-0585-4d1f-956f-45b6cf6d6e7a","Type":"ContainerStarted","Data":"2cb0b0c1f747d9d491d854f0a487cd72edfcc16f03223ee7553ffe3f64d6c55a"}
Sep 15 21:39:48.632285 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:48.631927    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"
Sep 15 21:39:48.633528 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:48.633507    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4" event={"ID":"45e087cc-6e58-42c6-9905-c1a045a35399","Type":"ContainerStarted","Data":"05dabfec53072c54015548c2eb7013696ef51107ca3dc82f90fb21ab561aceb0"}
Sep 15 21:39:48.633650 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:48.633549    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4" event={"ID":"45e087cc-6e58-42c6-9905-c1a045a35399","Type":"ContainerStarted","Data":"85338f894b0041af9cb6cbe6201259e828edcad0dae0c60cbedf4b25f871f4cb"}
Sep 15 21:39:48.633785 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:48.633769    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"
Sep 15 21:39:48.648600 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:48.648548    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6" podStartSLOduration=1.648531532 podStartE2EDuration="1.648531532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:48.645839177 +0000 UTC m=+826.457847624" watchObservedRunningTime="2026-09-15 21:39:48.648531532 +0000 UTC m=+826.460539978"
Sep 15 21:39:48.662629 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:48.662569    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4" podStartSLOduration=1.6625497889999998 podStartE2EDuration="1.662549789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:48.661556836 +0000 UTC m=+826.473565279" watchObservedRunningTime="2026-09-15 21:39:48.662549789 +0000 UTC m=+826.474558235"
Sep 15 21:39:49.572109 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:49.572078    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"
Sep 15 21:39:49.572314 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:49.572167    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"
Sep 15 21:39:52.557621 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:52.557584    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:39:55.810206 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:55.810175    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"]
Sep 15 21:39:55.815204 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:55.815181    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"
Sep 15 21:39:55.817375 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:55.817319    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a2d9c4\""
Sep 15 21:39:55.817520 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:55.817432    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a2d9c4-dockercfg-r9cqn\""
Sep 15 21:39:55.825626 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:55.825595    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"]
Sep 15 21:39:55.884592 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:55.884559    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpx8c\" (UniqueName: \"kubernetes.io/projected/3682aab2-51bd-41bf-adbd-ac92d55fc492-kube-api-access-qpx8c\") pod \"payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq\" (UID: \"3682aab2-51bd-41bf-adbd-ac92d55fc492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"
Sep 15 21:39:55.884800 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:55.884709    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3682aab2-51bd-41bf-adbd-ac92d55fc492-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq\" (UID: \"3682aab2-51bd-41bf-adbd-ac92d55fc492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"
Sep 15 21:39:55.986010 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:55.985972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3682aab2-51bd-41bf-adbd-ac92d55fc492-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq\" (UID: \"3682aab2-51bd-41bf-adbd-ac92d55fc492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"
Sep 15 21:39:55.986010 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:55.986026    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpx8c\" (UniqueName: \"kubernetes.io/projected/3682aab2-51bd-41bf-adbd-ac92d55fc492-kube-api-access-qpx8c\") pod \"payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq\" (UID: \"3682aab2-51bd-41bf-adbd-ac92d55fc492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"
Sep 15 21:39:55.986954 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:55.986922    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3682aab2-51bd-41bf-adbd-ac92d55fc492-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq\" (UID: \"3682aab2-51bd-41bf-adbd-ac92d55fc492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"
Sep 15 21:39:55.994146 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:55.994120    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpx8c\" (UniqueName: \"kubernetes.io/projected/3682aab2-51bd-41bf-adbd-ac92d55fc492-kube-api-access-qpx8c\") pod \"payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq\" (UID: \"3682aab2-51bd-41bf-adbd-ac92d55fc492\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"
Sep 15 21:39:56.129935 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:56.129902    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"
Sep 15 21:39:56.426697 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:56.426672    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"]
Sep 15 21:39:56.428944 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:39:56.428903    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3682aab2_51bd_41bf_adbd_ac92d55fc492.slice/crio-99e3fe5122a2cf29e6615ee6e5aa0549b6aa3ad8bc18ed16dc229bbed3f8ee78 WatchSource:0}: Error finding container 99e3fe5122a2cf29e6615ee6e5aa0549b6aa3ad8bc18ed16dc229bbed3f8ee78: Status 404 returned error can't find the container with id 99e3fe5122a2cf29e6615ee6e5aa0549b6aa3ad8bc18ed16dc229bbed3f8ee78
Sep 15 21:39:56.697286 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:56.697203    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq" event={"ID":"3682aab2-51bd-41bf-adbd-ac92d55fc492","Type":"ContainerStarted","Data":"4ecd9f9476ead038159c6a94aff4424ba5250422a739d7fe868ecc8d042f5a63"}
Sep 15 21:39:56.697576 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:56.697556    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"
Sep 15 21:39:56.697670 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:56.697583    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq" event={"ID":"3682aab2-51bd-41bf-adbd-ac92d55fc492","Type":"ContainerStarted","Data":"99e3fe5122a2cf29e6615ee6e5aa0549b6aa3ad8bc18ed16dc229bbed3f8ee78"}
Sep 15 21:39:56.713938 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:56.713874    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq" podStartSLOduration=1.713859008 podStartE2EDuration="1.713859008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:39:56.711933227 +0000 UTC m=+834.523941688" watchObservedRunningTime="2026-09-15 21:39:56.713859008 +0000 UTC m=+834.525867452"
Sep 15 21:39:58.635104 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:58.635068    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"
Sep 15 21:39:58.635546 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:58.635489    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"
Sep 15 21:39:59.641186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:59.641153    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"
Sep 15 21:39:59.641567 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:39:59.641240    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"
Sep 15 21:40:00.237954 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:00.237921    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj"]
Sep 15 21:40:00.241741 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:00.241725    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj"
Sep 15 21:40:00.243876 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:00.243852    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 21:40:00.243976 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:00.243894    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:40:00.248347 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:00.248323    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj"]
Sep 15 21:40:00.332796 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:00.332738    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2516bfb0-f883-4aea-a14e-d843f7f1f64d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj\" (UID: \"2516bfb0-f883-4aea-a14e-d843f7f1f64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj"
Sep 15 21:40:00.433606 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:00.433564    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2516bfb0-f883-4aea-a14e-d843f7f1f64d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj\" (UID: \"2516bfb0-f883-4aea-a14e-d843f7f1f64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj"
Sep 15 21:40:00.434288 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:00.434264    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2516bfb0-f883-4aea-a14e-d843f7f1f64d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj\" (UID: \"2516bfb0-f883-4aea-a14e-d843f7f1f64d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj"
Sep 15 21:40:00.555020 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:00.554922    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj"
Sep 15 21:40:00.839545 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:00.839523    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj"]
Sep 15 21:40:00.841888 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:40:00.841859    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2516bfb0_f883_4aea_a14e_d843f7f1f64d.slice/crio-2489ee87c165aea2e91b1399dfc8aa6d53e984d54e4c9a22a05ec64c43406cd7 WatchSource:0}: Error finding container 2489ee87c165aea2e91b1399dfc8aa6d53e984d54e4c9a22a05ec64c43406cd7: Status 404 returned error can't find the container with id 2489ee87c165aea2e91b1399dfc8aa6d53e984d54e4c9a22a05ec64c43406cd7
Sep 15 21:40:01.725928 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:01.725886    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj" event={"ID":"2516bfb0-f883-4aea-a14e-d843f7f1f64d","Type":"ContainerStarted","Data":"2489ee87c165aea2e91b1399dfc8aa6d53e984d54e4c9a22a05ec64c43406cd7"}
Sep 15 21:40:04.743794 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:04.743746    2575 generic.go:356] "Generic (PLEG): container finished" podID="76b8df82-039f-46ea-8800-4f9599f6916a" containerID="cd7f45acbcac78b69230c34cb9de40aa983c7b42cbc4d592a230b653fc41bd4f" exitCode=0
Sep 15 21:40:04.745473 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:04.745446    2575 generic.go:356] "Generic (PLEG): container finished" podID="2516bfb0-f883-4aea-a14e-d843f7f1f64d" containerID="87c39467f9ab4d4e98447b8cdd25e62cdd0f3e47791cec2b0c2b4425ca887fb2" exitCode=0
Sep 15 21:40:04.747731 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:04.747703    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7" event={"ID":"76b8df82-039f-46ea-8800-4f9599f6916a","Type":"ContainerDied","Data":"cd7f45acbcac78b69230c34cb9de40aa983c7b42cbc4d592a230b653fc41bd4f"}
Sep 15 21:40:04.747859 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:04.747777    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj" event={"ID":"2516bfb0-f883-4aea-a14e-d843f7f1f64d","Type":"ContainerDied","Data":"87c39467f9ab4d4e98447b8cdd25e62cdd0f3e47791cec2b0c2b4425ca887fb2"}
Sep 15 21:40:04.748167 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:04.748151    2575 scope.go:117] "RemoveContainer" containerID="cd7f45acbcac78b69230c34cb9de40aa983c7b42cbc4d592a230b653fc41bd4f"
Sep 15 21:40:04.749311 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:04.749285    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:40:04.749394 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:04.749350    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:40:04.749394 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:04.749387    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:40:05.754560 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:05.754515    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7" event={"ID":"76b8df82-039f-46ea-8800-4f9599f6916a","Type":"ContainerStarted","Data":"115a5bf847b4452d435048f40944575b446685184d3f0c046b94813f2933194b"}
Sep 15 21:40:05.928056 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:05.928021    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:40:05.928407 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:05.928381    2575 patch_prober.go:28] interesting pod/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 15 21:40:05.928492 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:05.928431    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7" podUID="76b8df82-039f-46ea-8800-4f9599f6916a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 15 21:40:05.935661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:05.935642    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj"
Sep 15 21:40:05.993484 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:05.993404    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2516bfb0-f883-4aea-a14e-d843f7f1f64d-maas-api-service-ca\") pod \"2516bfb0-f883-4aea-a14e-d843f7f1f64d\" (UID: \"2516bfb0-f883-4aea-a14e-d843f7f1f64d\") "
Sep 15 21:40:05.993831 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:05.993808    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2516bfb0-f883-4aea-a14e-d843f7f1f64d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2516bfb0-f883-4aea-a14e-d843f7f1f64d" (UID: "2516bfb0-f883-4aea-a14e-d843f7f1f64d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:40:06.094574 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:06.094544    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2516bfb0-f883-4aea-a14e-d843f7f1f64d-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:06.763989 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:06.763898    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj" event={"ID":"2516bfb0-f883-4aea-a14e-d843f7f1f64d","Type":"ContainerDied","Data":"2489ee87c165aea2e91b1399dfc8aa6d53e984d54e4c9a22a05ec64c43406cd7"}
Sep 15 21:40:06.763989 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:06.763962    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2489ee87c165aea2e91b1399dfc8aa6d53e984d54e4c9a22a05ec64c43406cd7"
Sep 15 21:40:06.764613 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:06.764595    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj"
Sep 15 21:40:06.927858 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:06.927822    2575 patch_prober.go:28] interesting pod/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 15 21:40:06.928031 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:06.927888    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7" podUID="76b8df82-039f-46ea-8800-4f9599f6916a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 15 21:40:07.705571 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:07.705542    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"
Sep 15 21:40:07.928225 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:07.928178    2575 patch_prober.go:28] interesting pod/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 15 21:40:07.928689 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:07.928243    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7" podUID="76b8df82-039f-46ea-8800-4f9599f6916a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 15 21:40:08.928270 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:08.928234    2575 patch_prober.go:28] interesting pod/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 15 21:40:08.928643 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:08.928307    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7" podUID="76b8df82-039f-46ea-8800-4f9599f6916a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 15 21:40:09.928025 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:09.927990    2575 patch_prober.go:28] interesting pod/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 15 21:40:09.928209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:09.928052    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7" podUID="76b8df82-039f-46ea-8800-4f9599f6916a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 15 21:40:10.928480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:10.928448    2575 patch_prober.go:28] interesting pod/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 15 21:40:10.928901 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:10.928505    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7" podUID="76b8df82-039f-46ea-8800-4f9599f6916a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 15 21:40:11.065174 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.065140    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"]
Sep 15 21:40:11.065413 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.065387    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4" podUID="45e087cc-6e58-42c6-9905-c1a045a35399" containerName="payload-processing" containerID="cri-o://05dabfec53072c54015548c2eb7013696ef51107ca3dc82f90fb21ab561aceb0" gracePeriod=30
Sep 15 21:40:11.071578 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.071540    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"]
Sep 15 21:40:11.072650 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.072620    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6" podUID="767add36-0585-4d1f-956f-45b6cf6d6e7a" containerName="payload-pre-processing" containerID="cri-o://be59937f44ae47b227a150df0f19b351a5069df58154843232a52f1b29c80e29" gracePeriod=30
Sep 15 21:40:11.225256 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.225178    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj"]
Sep 15 21:40:11.227225 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.227195    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3ba8d384-vqphj"]
Sep 15 21:40:11.323252 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.323229    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"
Sep 15 21:40:11.353688 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.353639    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"]
Sep 15 21:40:11.353947 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.353925    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7" podUID="76b8df82-039f-46ea-8800-4f9599f6916a" containerName="istio-proxy" containerID="cri-o://115a5bf847b4452d435048f40944575b446685184d3f0c046b94813f2933194b" gracePeriod=30
Sep 15 21:40:11.356097 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.356067    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"
Sep 15 21:40:11.446664 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.446583    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzgfs\" (UniqueName: \"kubernetes.io/projected/45e087cc-6e58-42c6-9905-c1a045a35399-kube-api-access-wzgfs\") pod \"45e087cc-6e58-42c6-9905-c1a045a35399\" (UID: \"45e087cc-6e58-42c6-9905-c1a045a35399\") "
Sep 15 21:40:11.446664 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.446644    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767add36-0585-4d1f-956f-45b6cf6d6e7a-plugins-config-volume\") pod \"767add36-0585-4d1f-956f-45b6cf6d6e7a\" (UID: \"767add36-0585-4d1f-956f-45b6cf6d6e7a\") "
Sep 15 21:40:11.446893 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.446822    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e087cc-6e58-42c6-9905-c1a045a35399-plugins-config-volume\") pod \"45e087cc-6e58-42c6-9905-c1a045a35399\" (UID: \"45e087cc-6e58-42c6-9905-c1a045a35399\") "
Sep 15 21:40:11.446893 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.446886    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qklfn\" (UniqueName: \"kubernetes.io/projected/767add36-0585-4d1f-956f-45b6cf6d6e7a-kube-api-access-qklfn\") pod \"767add36-0585-4d1f-956f-45b6cf6d6e7a\" (UID: \"767add36-0585-4d1f-956f-45b6cf6d6e7a\") "
Sep 15 21:40:11.447011 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.446987    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/767add36-0585-4d1f-956f-45b6cf6d6e7a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "767add36-0585-4d1f-956f-45b6cf6d6e7a" (UID: "767add36-0585-4d1f-956f-45b6cf6d6e7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:40:11.447171 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.447151    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e087cc-6e58-42c6-9905-c1a045a35399-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45e087cc-6e58-42c6-9905-c1a045a35399" (UID: "45e087cc-6e58-42c6-9905-c1a045a35399"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:40:11.447292 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.447275    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/767add36-0585-4d1f-956f-45b6cf6d6e7a-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:11.447348 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.447297    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45e087cc-6e58-42c6-9905-c1a045a35399-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:11.448987 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.448964    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e087cc-6e58-42c6-9905-c1a045a35399-kube-api-access-wzgfs" (OuterVolumeSpecName: "kube-api-access-wzgfs") pod "45e087cc-6e58-42c6-9905-c1a045a35399" (UID: "45e087cc-6e58-42c6-9905-c1a045a35399"). InnerVolumeSpecName "kube-api-access-wzgfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:40:11.449037 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.448965    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/767add36-0585-4d1f-956f-45b6cf6d6e7a-kube-api-access-qklfn" (OuterVolumeSpecName: "kube-api-access-qklfn") pod "767add36-0585-4d1f-956f-45b6cf6d6e7a" (UID: "767add36-0585-4d1f-956f-45b6cf6d6e7a"). InnerVolumeSpecName "kube-api-access-qklfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:40:11.548845 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.548812    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzgfs\" (UniqueName: \"kubernetes.io/projected/45e087cc-6e58-42c6-9905-c1a045a35399-kube-api-access-wzgfs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:11.548845 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.548839    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qklfn\" (UniqueName: \"kubernetes.io/projected/767add36-0585-4d1f-956f-45b6cf6d6e7a-kube-api-access-qklfn\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:11.800420 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.800336    2575 generic.go:356] "Generic (PLEG): container finished" podID="45e087cc-6e58-42c6-9905-c1a045a35399" containerID="05dabfec53072c54015548c2eb7013696ef51107ca3dc82f90fb21ab561aceb0" exitCode=0
Sep 15 21:40:11.800420 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.800396    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"
Sep 15 21:40:11.800648 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.800417    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4" event={"ID":"45e087cc-6e58-42c6-9905-c1a045a35399","Type":"ContainerDied","Data":"05dabfec53072c54015548c2eb7013696ef51107ca3dc82f90fb21ab561aceb0"}
Sep 15 21:40:11.800648 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.800479    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4" event={"ID":"45e087cc-6e58-42c6-9905-c1a045a35399","Type":"ContainerDied","Data":"85338f894b0041af9cb6cbe6201259e828edcad0dae0c60cbedf4b25f871f4cb"}
Sep 15 21:40:11.800648 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.800504    2575 scope.go:117] "RemoveContainer" containerID="05dabfec53072c54015548c2eb7013696ef51107ca3dc82f90fb21ab561aceb0"
Sep 15 21:40:11.801957 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.801933    2575 generic.go:356] "Generic (PLEG): container finished" podID="767add36-0585-4d1f-956f-45b6cf6d6e7a" containerID="be59937f44ae47b227a150df0f19b351a5069df58154843232a52f1b29c80e29" exitCode=0
Sep 15 21:40:11.802091 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.801972    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"
Sep 15 21:40:11.802091 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.802017    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6" event={"ID":"767add36-0585-4d1f-956f-45b6cf6d6e7a","Type":"ContainerDied","Data":"be59937f44ae47b227a150df0f19b351a5069df58154843232a52f1b29c80e29"}
Sep 15 21:40:11.802091 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.802061    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6" event={"ID":"767add36-0585-4d1f-956f-45b6cf6d6e7a","Type":"ContainerDied","Data":"2cb0b0c1f747d9d491d854f0a487cd72edfcc16f03223ee7553ffe3f64d6c55a"}
Sep 15 21:40:11.810794 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.810774    2575 scope.go:117] "RemoveContainer" containerID="05dabfec53072c54015548c2eb7013696ef51107ca3dc82f90fb21ab561aceb0"
Sep 15 21:40:11.811178 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:40:11.811092    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05dabfec53072c54015548c2eb7013696ef51107ca3dc82f90fb21ab561aceb0\": container with ID starting with 05dabfec53072c54015548c2eb7013696ef51107ca3dc82f90fb21ab561aceb0 not found: ID does not exist" containerID="05dabfec53072c54015548c2eb7013696ef51107ca3dc82f90fb21ab561aceb0"
Sep 15 21:40:11.811178 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.811126    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05dabfec53072c54015548c2eb7013696ef51107ca3dc82f90fb21ab561aceb0"} err="failed to get container status \"05dabfec53072c54015548c2eb7013696ef51107ca3dc82f90fb21ab561aceb0\": rpc error: code = NotFound desc = could not find container \"05dabfec53072c54015548c2eb7013696ef51107ca3dc82f90fb21ab561aceb0\": container with ID starting with 05dabfec53072c54015548c2eb7013696ef51107ca3dc82f90fb21ab561aceb0 not found: ID does not exist"
Sep 15 21:40:11.811178 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.811148    2575 scope.go:117] "RemoveContainer" containerID="be59937f44ae47b227a150df0f19b351a5069df58154843232a52f1b29c80e29"
Sep 15 21:40:11.819984 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.819964    2575 scope.go:117] "RemoveContainer" containerID="be59937f44ae47b227a150df0f19b351a5069df58154843232a52f1b29c80e29"
Sep 15 21:40:11.820306 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:40:11.820286    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be59937f44ae47b227a150df0f19b351a5069df58154843232a52f1b29c80e29\": container with ID starting with be59937f44ae47b227a150df0f19b351a5069df58154843232a52f1b29c80e29 not found: ID does not exist" containerID="be59937f44ae47b227a150df0f19b351a5069df58154843232a52f1b29c80e29"
Sep 15 21:40:11.820388 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.820312    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be59937f44ae47b227a150df0f19b351a5069df58154843232a52f1b29c80e29"} err="failed to get container status \"be59937f44ae47b227a150df0f19b351a5069df58154843232a52f1b29c80e29\": rpc error: code = NotFound desc = could not find container \"be59937f44ae47b227a150df0f19b351a5069df58154843232a52f1b29c80e29\": container with ID starting with be59937f44ae47b227a150df0f19b351a5069df58154843232a52f1b29c80e29 not found: ID does not exist"
Sep 15 21:40:11.832242 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.832202    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"]
Sep 15 21:40:11.834485 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.834465    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3ba8d384-5c7c48bd4b-274g6"]
Sep 15 21:40:11.845581 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.845558    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"]
Sep 15 21:40:11.851894 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:11.851872    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3ba8d384-7774f9cfcc-pk7d4"]
Sep 15 21:40:12.296486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.296457    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"]
Sep 15 21:40:12.296940 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.296925    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2516bfb0-f883-4aea-a14e-d843f7f1f64d" containerName="revoke-keys"
Sep 15 21:40:12.296987 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.296943    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2516bfb0-f883-4aea-a14e-d843f7f1f64d" containerName="revoke-keys"
Sep 15 21:40:12.296987 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.296955    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45e087cc-6e58-42c6-9905-c1a045a35399" containerName="payload-processing"
Sep 15 21:40:12.296987 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.296961    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="45e087cc-6e58-42c6-9905-c1a045a35399" containerName="payload-processing"
Sep 15 21:40:12.296987 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.296974    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="767add36-0585-4d1f-956f-45b6cf6d6e7a" containerName="payload-pre-processing"
Sep 15 21:40:12.296987 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.296980    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="767add36-0585-4d1f-956f-45b6cf6d6e7a" containerName="payload-pre-processing"
Sep 15 21:40:12.297138 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.297045    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="767add36-0585-4d1f-956f-45b6cf6d6e7a" containerName="payload-pre-processing"
Sep 15 21:40:12.297138 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.297055    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="45e087cc-6e58-42c6-9905-c1a045a35399" containerName="payload-processing"
Sep 15 21:40:12.297138 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.297064    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2516bfb0-f883-4aea-a14e-d843f7f1f64d" containerName="revoke-keys"
Sep 15 21:40:12.301611 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.301594    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.303800 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.303780    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ee52fdef-gw-openshift-default-dockercfg-fgqck\""
Sep 15 21:40:12.315885 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.315865    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"]
Sep 15 21:40:12.356278 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.356244    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-workload-certs\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.356278 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.356275    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af89c3ce-6dda-46ea-8962-424a22d9ba36-istiod-ca-cert\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.356462 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.356304    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-credential-socket\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.356462 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.356321    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-token\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.356462 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.356411    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-podinfo\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.356462 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.356436    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krb85\" (UniqueName: \"kubernetes.io/projected/af89c3ce-6dda-46ea-8962-424a22d9ba36-kube-api-access-krb85\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.356671 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.356536    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-workload-socket\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.356671 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.356600    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-data\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.356671 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.356648    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-envoy\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.457776 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.457727    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-workload-certs\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.457981 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.457791    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af89c3ce-6dda-46ea-8962-424a22d9ba36-istiod-ca-cert\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.457981 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.457833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-credential-socket\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.457981 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.457855    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-token\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.458167 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.458029    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-podinfo\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.458167 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.458070    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krb85\" (UniqueName: \"kubernetes.io/projected/af89c3ce-6dda-46ea-8962-424a22d9ba36-kube-api-access-krb85\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.458167 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.458141    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-workload-certs\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.458167 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.458156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-workload-socket\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.458351 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.458208    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-credential-socket\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.458351 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.458277    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-data\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.458351 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.458324    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-envoy\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.458508 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.458398    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-workload-socket\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.458619 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.458596    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af89c3ce-6dda-46ea-8962-424a22d9ba36-istiod-ca-cert\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.458674 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.458618    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-data\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.460560 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.460536    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-envoy\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.460808 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.460792    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-podinfo\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.468121 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.468097    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-token\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.468309 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.468295    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krb85\" (UniqueName: \"kubernetes.io/projected/af89c3ce-6dda-46ea-8962-424a22d9ba36-kube-api-access-krb85\") pod \"e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") " pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.612039 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.611940    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:12.747776 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.747725    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2516bfb0-f883-4aea-a14e-d843f7f1f64d" path="/var/lib/kubelet/pods/2516bfb0-f883-4aea-a14e-d843f7f1f64d/volumes"
Sep 15 21:40:12.748235 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.748214    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45e087cc-6e58-42c6-9905-c1a045a35399" path="/var/lib/kubelet/pods/45e087cc-6e58-42c6-9905-c1a045a35399/volumes"
Sep 15 21:40:12.748593 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.748575    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="767add36-0585-4d1f-956f-45b6cf6d6e7a" path="/var/lib/kubelet/pods/767add36-0585-4d1f-956f-45b6cf6d6e7a/volumes"
Sep 15 21:40:12.887350 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.887323    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"]
Sep 15 21:40:12.889065 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:40:12.889039    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf89c3ce_6dda_46ea_8962_424a22d9ba36.slice/crio-bba42cd897e8b19be1237ab2027c47d1eb458493ef3e350dee445e6ef6f6c2f1 WatchSource:0}: Error finding container bba42cd897e8b19be1237ab2027c47d1eb458493ef3e350dee445e6ef6f6c2f1: Status 404 returned error can't find the container with id bba42cd897e8b19be1237ab2027c47d1eb458493ef3e350dee445e6ef6f6c2f1
Sep 15 21:40:12.891507 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.891478    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:40:12.891595 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.891542    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:40:12.891595 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:12.891578    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:40:13.814957 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:13.814917    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl" event={"ID":"af89c3ce-6dda-46ea-8962-424a22d9ba36","Type":"ContainerStarted","Data":"bae3cb1d52e700be0b691d94227181dfb70dfa25d6dec3970e28bf417de87293"}
Sep 15 21:40:13.815395 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:13.814966    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl" event={"ID":"af89c3ce-6dda-46ea-8962-424a22d9ba36","Type":"ContainerStarted","Data":"bba42cd897e8b19be1237ab2027c47d1eb458493ef3e350dee445e6ef6f6c2f1"}
Sep 15 21:40:13.835342 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:13.835278    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl" podStartSLOduration=1.835256644 podStartE2EDuration="1.835256644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:40:13.833005047 +0000 UTC m=+851.645013491" watchObservedRunningTime="2026-09-15 21:40:13.835256644 +0000 UTC m=+851.647265088"
Sep 15 21:40:14.608227 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:14.608192    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:40:14.612414 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:14.612391    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:14.618076 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:14.618054    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:14.819932 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:14.819894    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:14.820961 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:14.820945    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:14.935895 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:14.935808    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:40:16.607218 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.607191    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:40:16.704454 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.704354    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8st8f\" (UniqueName: \"kubernetes.io/projected/76b8df82-039f-46ea-8800-4f9599f6916a-kube-api-access-8st8f\") pod \"76b8df82-039f-46ea-8800-4f9599f6916a\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") "
Sep 15 21:40:16.704454 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.704404    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b8df82-039f-46ea-8800-4f9599f6916a-istio-podinfo\") pod \"76b8df82-039f-46ea-8800-4f9599f6916a\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") "
Sep 15 21:40:16.704686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.704459    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-istio-data\") pod \"76b8df82-039f-46ea-8800-4f9599f6916a\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") "
Sep 15 21:40:16.704686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.704488    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-workload-socket\") pod \"76b8df82-039f-46ea-8800-4f9599f6916a\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") "
Sep 15 21:40:16.704686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.704513    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-credential-socket\") pod \"76b8df82-039f-46ea-8800-4f9599f6916a\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") "
Sep 15 21:40:16.704686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.704556    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-istio-envoy\") pod \"76b8df82-039f-46ea-8800-4f9599f6916a\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") "
Sep 15 21:40:16.704686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.704578    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b8df82-039f-46ea-8800-4f9599f6916a-istio-token\") pod \"76b8df82-039f-46ea-8800-4f9599f6916a\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") "
Sep 15 21:40:16.704686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.704641    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b8df82-039f-46ea-8800-4f9599f6916a-istiod-ca-cert\") pod \"76b8df82-039f-46ea-8800-4f9599f6916a\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") "
Sep 15 21:40:16.705003 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.704709    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-workload-certs\") pod \"76b8df82-039f-46ea-8800-4f9599f6916a\" (UID: \"76b8df82-039f-46ea-8800-4f9599f6916a\") "
Sep 15 21:40:16.705003 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.704827    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-istio-data" (OuterVolumeSpecName: "istio-data") pod "76b8df82-039f-46ea-8800-4f9599f6916a" (UID: "76b8df82-039f-46ea-8800-4f9599f6916a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:16.705003 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.704832    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "76b8df82-039f-46ea-8800-4f9599f6916a" (UID: "76b8df82-039f-46ea-8800-4f9599f6916a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:16.705003 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.704861    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "76b8df82-039f-46ea-8800-4f9599f6916a" (UID: "76b8df82-039f-46ea-8800-4f9599f6916a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:16.705192 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.705112    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "76b8df82-039f-46ea-8800-4f9599f6916a" (UID: "76b8df82-039f-46ea-8800-4f9599f6916a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:16.705192 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.705135    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:16.705192 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.705163    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:16.705192 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.705177    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:16.705192 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.705135    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76b8df82-039f-46ea-8800-4f9599f6916a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "76b8df82-039f-46ea-8800-4f9599f6916a" (UID: "76b8df82-039f-46ea-8800-4f9599f6916a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:40:16.706988 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.706954    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b8df82-039f-46ea-8800-4f9599f6916a-istio-token" (OuterVolumeSpecName: "istio-token") pod "76b8df82-039f-46ea-8800-4f9599f6916a" (UID: "76b8df82-039f-46ea-8800-4f9599f6916a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:40:16.706988 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.706971    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "76b8df82-039f-46ea-8800-4f9599f6916a" (UID: "76b8df82-039f-46ea-8800-4f9599f6916a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:16.707213 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.707198    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76b8df82-039f-46ea-8800-4f9599f6916a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "76b8df82-039f-46ea-8800-4f9599f6916a" (UID: "76b8df82-039f-46ea-8800-4f9599f6916a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:40:16.707255 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.707226    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b8df82-039f-46ea-8800-4f9599f6916a-kube-api-access-8st8f" (OuterVolumeSpecName: "kube-api-access-8st8f") pod "76b8df82-039f-46ea-8800-4f9599f6916a" (UID: "76b8df82-039f-46ea-8800-4f9599f6916a"). InnerVolumeSpecName "kube-api-access-8st8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:40:16.806005 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.805973    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76b8df82-039f-46ea-8800-4f9599f6916a-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:16.806005 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.806000    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:16.806005 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.806009    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8st8f\" (UniqueName: \"kubernetes.io/projected/76b8df82-039f-46ea-8800-4f9599f6916a-kube-api-access-8st8f\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:16.806240 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.806018    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76b8df82-039f-46ea-8800-4f9599f6916a-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:16.806240 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.806027    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76b8df82-039f-46ea-8800-4f9599f6916a-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:16.806240 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.806036    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76b8df82-039f-46ea-8800-4f9599f6916a-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:16.832950 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.832912    2575 generic.go:356] "Generic (PLEG): container finished" podID="76b8df82-039f-46ea-8800-4f9599f6916a" containerID="115a5bf847b4452d435048f40944575b446685184d3f0c046b94813f2933194b" exitCode=0
Sep 15 21:40:16.833095 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.832963    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"
Sep 15 21:40:16.833095 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.833003    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7" event={"ID":"76b8df82-039f-46ea-8800-4f9599f6916a","Type":"ContainerDied","Data":"115a5bf847b4452d435048f40944575b446685184d3f0c046b94813f2933194b"}
Sep 15 21:40:16.833095 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.833050    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7" event={"ID":"76b8df82-039f-46ea-8800-4f9599f6916a","Type":"ContainerDied","Data":"9822c88ab11d4b7ecb3658ea7ad0abba6cf2f4bc783a11f39d8cee6543f712f4"}
Sep 15 21:40:16.833095 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.833071    2575 scope.go:117] "RemoveContainer" containerID="115a5bf847b4452d435048f40944575b446685184d3f0c046b94813f2933194b"
Sep 15 21:40:16.842182 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.842163    2575 scope.go:117] "RemoveContainer" containerID="cd7f45acbcac78b69230c34cb9de40aa983c7b42cbc4d592a230b653fc41bd4f"
Sep 15 21:40:16.848984 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.848961    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"]
Sep 15 21:40:16.851360 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.851342    2575 scope.go:117] "RemoveContainer" containerID="115a5bf847b4452d435048f40944575b446685184d3f0c046b94813f2933194b"
Sep 15 21:40:16.851662 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:40:16.851644    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115a5bf847b4452d435048f40944575b446685184d3f0c046b94813f2933194b\": container with ID starting with 115a5bf847b4452d435048f40944575b446685184d3f0c046b94813f2933194b not found: ID does not exist" containerID="115a5bf847b4452d435048f40944575b446685184d3f0c046b94813f2933194b"
Sep 15 21:40:16.851702 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.851671    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115a5bf847b4452d435048f40944575b446685184d3f0c046b94813f2933194b"} err="failed to get container status \"115a5bf847b4452d435048f40944575b446685184d3f0c046b94813f2933194b\": rpc error: code = NotFound desc = could not find container \"115a5bf847b4452d435048f40944575b446685184d3f0c046b94813f2933194b\": container with ID starting with 115a5bf847b4452d435048f40944575b446685184d3f0c046b94813f2933194b not found: ID does not exist"
Sep 15 21:40:16.851702 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.851690    2575 scope.go:117] "RemoveContainer" containerID="cd7f45acbcac78b69230c34cb9de40aa983c7b42cbc4d592a230b653fc41bd4f"
Sep 15 21:40:16.852001 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:40:16.851983    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd7f45acbcac78b69230c34cb9de40aa983c7b42cbc4d592a230b653fc41bd4f\": container with ID starting with cd7f45acbcac78b69230c34cb9de40aa983c7b42cbc4d592a230b653fc41bd4f not found: ID does not exist" containerID="cd7f45acbcac78b69230c34cb9de40aa983c7b42cbc4d592a230b653fc41bd4f"
Sep 15 21:40:16.852046 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.852010    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd7f45acbcac78b69230c34cb9de40aa983c7b42cbc4d592a230b653fc41bd4f"} err="failed to get container status \"cd7f45acbcac78b69230c34cb9de40aa983c7b42cbc4d592a230b653fc41bd4f\": rpc error: code = NotFound desc = could not find container \"cd7f45acbcac78b69230c34cb9de40aa983c7b42cbc4d592a230b653fc41bd4f\": container with ID starting with cd7f45acbcac78b69230c34cb9de40aa983c7b42cbc4d592a230b653fc41bd4f not found: ID does not exist"
Sep 15 21:40:16.855678 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:16.855655    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3ba8d384-openshift-default-85857658f4-xd9b7"]
Sep 15 21:40:17.076508 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.076423    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"]
Sep 15 21:40:17.076882 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.076869    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b8df82-039f-46ea-8800-4f9599f6916a" containerName="istio-proxy"
Sep 15 21:40:17.076945 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.076884    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b8df82-039f-46ea-8800-4f9599f6916a" containerName="istio-proxy"
Sep 15 21:40:17.076945 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.076897    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b8df82-039f-46ea-8800-4f9599f6916a" containerName="istio-proxy"
Sep 15 21:40:17.076945 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.076902    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b8df82-039f-46ea-8800-4f9599f6916a" containerName="istio-proxy"
Sep 15 21:40:17.077059 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.076978    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b8df82-039f-46ea-8800-4f9599f6916a" containerName="istio-proxy"
Sep 15 21:40:17.077059 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.076988    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b8df82-039f-46ea-8800-4f9599f6916a" containerName="istio-proxy"
Sep 15 21:40:17.079694 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.079678    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"
Sep 15 21:40:17.081724 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.081705    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-ee52fdef\""
Sep 15 21:40:17.081724 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.081718    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-ee52fdef-dockercfg-t7qz4\""
Sep 15 21:40:17.090448 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.090421    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"]
Sep 15 21:40:17.128523 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.128489    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"]
Sep 15 21:40:17.130947 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.130931    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"
Sep 15 21:40:17.142561 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.142534    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"]
Sep 15 21:40:17.210239 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.210205    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w57p6\" (UniqueName: \"kubernetes.io/projected/84495d20-9ad8-4b52-b8d6-777d4f055ee2-kube-api-access-w57p6\") pod \"payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx\" (UID: \"84495d20-9ad8-4b52-b8d6-777d4f055ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"
Sep 15 21:40:17.210421 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.210256    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d107b42-3da1-4437-b321-0669ce80dd00-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w\" (UID: \"7d107b42-3da1-4437-b321-0669ce80dd00\") " pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"
Sep 15 21:40:17.210421 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.210316    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84495d20-9ad8-4b52-b8d6-777d4f055ee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx\" (UID: \"84495d20-9ad8-4b52-b8d6-777d4f055ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"
Sep 15 21:40:17.210421 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.210350    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcs8c\" (UniqueName: \"kubernetes.io/projected/7d107b42-3da1-4437-b321-0669ce80dd00-kube-api-access-dcs8c\") pod \"payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w\" (UID: \"7d107b42-3da1-4437-b321-0669ce80dd00\") " pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"
Sep 15 21:40:17.311858 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.311819    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w57p6\" (UniqueName: \"kubernetes.io/projected/84495d20-9ad8-4b52-b8d6-777d4f055ee2-kube-api-access-w57p6\") pod \"payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx\" (UID: \"84495d20-9ad8-4b52-b8d6-777d4f055ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"
Sep 15 21:40:17.312041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.311865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d107b42-3da1-4437-b321-0669ce80dd00-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w\" (UID: \"7d107b42-3da1-4437-b321-0669ce80dd00\") " pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"
Sep 15 21:40:17.312041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.311902    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84495d20-9ad8-4b52-b8d6-777d4f055ee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx\" (UID: \"84495d20-9ad8-4b52-b8d6-777d4f055ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"
Sep 15 21:40:17.312041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.311928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcs8c\" (UniqueName: \"kubernetes.io/projected/7d107b42-3da1-4437-b321-0669ce80dd00-kube-api-access-dcs8c\") pod \"payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w\" (UID: \"7d107b42-3da1-4437-b321-0669ce80dd00\") " pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"
Sep 15 21:40:17.312912 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.312891    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84495d20-9ad8-4b52-b8d6-777d4f055ee2-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx\" (UID: \"84495d20-9ad8-4b52-b8d6-777d4f055ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"
Sep 15 21:40:17.312966 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.312891    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d107b42-3da1-4437-b321-0669ce80dd00-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w\" (UID: \"7d107b42-3da1-4437-b321-0669ce80dd00\") " pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"
Sep 15 21:40:17.319538 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.319508    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcs8c\" (UniqueName: \"kubernetes.io/projected/7d107b42-3da1-4437-b321-0669ce80dd00-kube-api-access-dcs8c\") pod \"payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w\" (UID: \"7d107b42-3da1-4437-b321-0669ce80dd00\") " pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"
Sep 15 21:40:17.319828 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.319811    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w57p6\" (UniqueName: \"kubernetes.io/projected/84495d20-9ad8-4b52-b8d6-777d4f055ee2-kube-api-access-w57p6\") pod \"payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx\" (UID: \"84495d20-9ad8-4b52-b8d6-777d4f055ee2\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"
Sep 15 21:40:17.391741 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.391702    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"
Sep 15 21:40:17.441808 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.441725    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"
Sep 15 21:40:17.667116 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.667085    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"]
Sep 15 21:40:17.669209 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:40:17.669179    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d107b42_3da1_4437_b321_0669ce80dd00.slice/crio-1f13340170ea812b628ed38dcca1a4d9b67aead24b7922d6cc1ac3e103e0456d WatchSource:0}: Error finding container 1f13340170ea812b628ed38dcca1a4d9b67aead24b7922d6cc1ac3e103e0456d: Status 404 returned error can't find the container with id 1f13340170ea812b628ed38dcca1a4d9b67aead24b7922d6cc1ac3e103e0456d
Sep 15 21:40:17.728090 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.728064    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"]
Sep 15 21:40:17.730688 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:40:17.730658    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84495d20_9ad8_4b52_b8d6_777d4f055ee2.slice/crio-dd1555241d9d9ce3ce4df43f2a1cbdb3967cc8328324ade2ff10403b8fa98897 WatchSource:0}: Error finding container dd1555241d9d9ce3ce4df43f2a1cbdb3967cc8328324ade2ff10403b8fa98897: Status 404 returned error can't find the container with id dd1555241d9d9ce3ce4df43f2a1cbdb3967cc8328324ade2ff10403b8fa98897
Sep 15 21:40:17.840837 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.840795    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w" event={"ID":"7d107b42-3da1-4437-b321-0669ce80dd00","Type":"ContainerStarted","Data":"381b7ab47d4533ee1b031c5c842a392fe661f91010e15b0c9182f004fb81b986"}
Sep 15 21:40:17.841040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.840858    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w" event={"ID":"7d107b42-3da1-4437-b321-0669ce80dd00","Type":"ContainerStarted","Data":"1f13340170ea812b628ed38dcca1a4d9b67aead24b7922d6cc1ac3e103e0456d"}
Sep 15 21:40:17.841040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.840943    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"
Sep 15 21:40:17.842421 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.842394    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx" event={"ID":"84495d20-9ad8-4b52-b8d6-777d4f055ee2","Type":"ContainerStarted","Data":"13ac18e569bb41aa086b90cd7aafb1835746c518ef631a9582d39aca6f55864d"}
Sep 15 21:40:17.842549 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.842439    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx" event={"ID":"84495d20-9ad8-4b52-b8d6-777d4f055ee2","Type":"ContainerStarted","Data":"dd1555241d9d9ce3ce4df43f2a1cbdb3967cc8328324ade2ff10403b8fa98897"}
Sep 15 21:40:17.842549 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.842487    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"
Sep 15 21:40:17.857718 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.857662    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w" podStartSLOduration=0.857645402 podStartE2EDuration="857.645402ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:40:17.855664204 +0000 UTC m=+855.667672653" watchObservedRunningTime="2026-09-15 21:40:17.857645402 +0000 UTC m=+855.669653846"
Sep 15 21:40:17.876265 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:17.876204    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx" podStartSLOduration=0.876182039 podStartE2EDuration="876.182039ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:40:17.873939134 +0000 UTC m=+855.685947581" watchObservedRunningTime="2026-09-15 21:40:17.876182039 +0000 UTC m=+855.688190484"
Sep 15 21:40:18.747724 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:18.747694    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76b8df82-039f-46ea-8800-4f9599f6916a" path="/var/lib/kubelet/pods/76b8df82-039f-46ea-8800-4f9599f6916a/volumes"
Sep 15 21:40:28.849830 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:28.849800    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"
Sep 15 21:40:28.850228 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:28.849868    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"
Sep 15 21:40:33.626223 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:33.626186    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:40:36.948705 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:36.948666    2575 generic.go:356] "Generic (PLEG): container finished" podID="af89c3ce-6dda-46ea-8962-424a22d9ba36" containerID="bae3cb1d52e700be0b691d94227181dfb70dfa25d6dec3970e28bf417de87293" exitCode=0
Sep 15 21:40:36.949137 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:36.948737    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl" event={"ID":"af89c3ce-6dda-46ea-8962-424a22d9ba36","Type":"ContainerDied","Data":"bae3cb1d52e700be0b691d94227181dfb70dfa25d6dec3970e28bf417de87293"}
Sep 15 21:40:36.949206 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:36.949191    2575 scope.go:117] "RemoveContainer" containerID="bae3cb1d52e700be0b691d94227181dfb70dfa25d6dec3970e28bf417de87293"
Sep 15 21:40:36.950283 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:36.950259    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:40:36.950337 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:36.950323    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:40:36.950373 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:36.950353    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:40:37.968687 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:37.968638    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl" event={"ID":"af89c3ce-6dda-46ea-8962-424a22d9ba36","Type":"ContainerStarted","Data":"e28abe7f2ea838689ae7941afb7406343fc38046b5646bacb3bd728046dd13b7"}
Sep 15 21:40:38.612368 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:38.612327    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:38.612582 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:38.612560    2575 patch_prober.go:28] interesting pod/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 15 21:40:38.612629 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:38.612609    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl" podUID="af89c3ce-6dda-46ea-8962-424a22d9ba36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 15 21:40:39.612596 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:39.612561    2575 patch_prober.go:28] interesting pod/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 15 21:40:39.613035 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:39.612618    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl" podUID="af89c3ce-6dda-46ea-8962-424a22d9ba36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 15 21:40:40.270780 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:40.270711    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp"]
Sep 15 21:40:40.274877 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:40.274848    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp"
Sep 15 21:40:40.277099 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:40.277073    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 21:40:40.277287 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:40.277269    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:40:40.282158 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:40.282135    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp"]
Sep 15 21:40:40.372781 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:40.372334    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c10ef0e9-e3c4-419f-80a1-2e5a973c7950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp\" (UID: \"c10ef0e9-e3c4-419f-80a1-2e5a973c7950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp"
Sep 15 21:40:40.473441 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:40.473402    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c10ef0e9-e3c4-419f-80a1-2e5a973c7950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp\" (UID: \"c10ef0e9-e3c4-419f-80a1-2e5a973c7950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp"
Sep 15 21:40:40.474443 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:40.474387    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c10ef0e9-e3c4-419f-80a1-2e5a973c7950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp\" (UID: \"c10ef0e9-e3c4-419f-80a1-2e5a973c7950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp"
Sep 15 21:40:40.590343 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:40.590256    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp"
Sep 15 21:40:40.612627 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:40.612589    2575 patch_prober.go:28] interesting pod/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 15 21:40:40.612954 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:40.612673    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl" podUID="af89c3ce-6dda-46ea-8962-424a22d9ba36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 15 21:40:40.879369 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:40.879335    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp"]
Sep 15 21:40:40.988320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:40.988283    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp" event={"ID":"c10ef0e9-e3c4-419f-80a1-2e5a973c7950","Type":"ContainerStarted","Data":"6a45c01582f2fa4391abd2556a1ab72d5123eeb4e52fe8fd0a1785a370becb4d"}
Sep 15 21:40:41.612859 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:41.612824    2575 patch_prober.go:28] interesting pod/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 15 21:40:41.613234 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:41.612880    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl" podUID="af89c3ce-6dda-46ea-8962-424a22d9ba36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 15 21:40:41.995589 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:41.995551    2575 generic.go:356] "Generic (PLEG): container finished" podID="c10ef0e9-e3c4-419f-80a1-2e5a973c7950" containerID="0b1e32f1ef822cf65f3de3bf4744f09f327c9029988f7f468e17fa2912f56cb4" exitCode=0
Sep 15 21:40:41.995737 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:41.995597    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp" event={"ID":"c10ef0e9-e3c4-419f-80a1-2e5a973c7950","Type":"ContainerDied","Data":"0b1e32f1ef822cf65f3de3bf4744f09f327c9029988f7f468e17fa2912f56cb4"}
Sep 15 21:40:42.612474 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:42.612439    2575 patch_prober.go:28] interesting pod/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Sep 15 21:40:42.612666 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:42.612491    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl" podUID="af89c3ce-6dda-46ea-8962-424a22d9ba36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Sep 15 21:40:43.166172 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:43.166144    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp"
Sep 15 21:40:43.206245 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:43.206206    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c10ef0e9-e3c4-419f-80a1-2e5a973c7950-maas-api-service-ca\") pod \"c10ef0e9-e3c4-419f-80a1-2e5a973c7950\" (UID: \"c10ef0e9-e3c4-419f-80a1-2e5a973c7950\") "
Sep 15 21:40:43.206520 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:43.206499    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10ef0e9-e3c4-419f-80a1-2e5a973c7950-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c10ef0e9-e3c4-419f-80a1-2e5a973c7950" (UID: "c10ef0e9-e3c4-419f-80a1-2e5a973c7950"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:40:43.206595 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:43.206577    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c10ef0e9-e3c4-419f-80a1-2e5a973c7950-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:43.617035 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:43.616996    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:43.617404 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:43.617379    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:43.617989 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:43.617975    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:44.008113 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:44.008036    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp"
Sep 15 21:40:44.008113 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:44.008074    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp" event={"ID":"c10ef0e9-e3c4-419f-80a1-2e5a973c7950","Type":"ContainerDied","Data":"6a45c01582f2fa4391abd2556a1ab72d5123eeb4e52fe8fd0a1785a370becb4d"}
Sep 15 21:40:44.008314 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:44.008118    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a45c01582f2fa4391abd2556a1ab72d5123eeb4e52fe8fd0a1785a370becb4d"
Sep 15 21:40:46.680059 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:46.680026    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"]
Sep 15 21:40:46.680493 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:46.680303    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w" podUID="7d107b42-3da1-4437-b321-0669ce80dd00" containerName="payload-processing" containerID="cri-o://381b7ab47d4533ee1b031c5c842a392fe661f91010e15b0c9182f004fb81b986" gracePeriod=30
Sep 15 21:40:46.694091 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:46.694059    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"]
Sep 15 21:40:46.694356 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:46.694282    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx" podUID="84495d20-9ad8-4b52-b8d6-777d4f055ee2" containerName="payload-pre-processing" containerID="cri-o://13ac18e569bb41aa086b90cd7aafb1835746c518ef631a9582d39aca6f55864d" gracePeriod=30
Sep 15 21:40:46.851338 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:46.851303    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp"]
Sep 15 21:40:46.853403 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:46.853379    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ee52fdef-27bhp"]
Sep 15 21:40:46.963286 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:46.963259    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"
Sep 15 21:40:46.990024 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:46.989996    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"]
Sep 15 21:40:46.990209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:46.990176    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl" podUID="af89c3ce-6dda-46ea-8962-424a22d9ba36" containerName="istio-proxy" containerID="cri-o://e28abe7f2ea838689ae7941afb7406343fc38046b5646bacb3bd728046dd13b7" gracePeriod=30
Sep 15 21:40:46.990845 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:46.990814    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"
Sep 15 21:40:47.025830 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.025800    2575 generic.go:356] "Generic (PLEG): container finished" podID="7d107b42-3da1-4437-b321-0669ce80dd00" containerID="381b7ab47d4533ee1b031c5c842a392fe661f91010e15b0c9182f004fb81b986" exitCode=0
Sep 15 21:40:47.025991 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.025851    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"
Sep 15 21:40:47.025991 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.025846    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w" event={"ID":"7d107b42-3da1-4437-b321-0669ce80dd00","Type":"ContainerDied","Data":"381b7ab47d4533ee1b031c5c842a392fe661f91010e15b0c9182f004fb81b986"}
Sep 15 21:40:47.026108 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.025985    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w" event={"ID":"7d107b42-3da1-4437-b321-0669ce80dd00","Type":"ContainerDied","Data":"1f13340170ea812b628ed38dcca1a4d9b67aead24b7922d6cc1ac3e103e0456d"}
Sep 15 21:40:47.026108 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.026024    2575 scope.go:117] "RemoveContainer" containerID="381b7ab47d4533ee1b031c5c842a392fe661f91010e15b0c9182f004fb81b986"
Sep 15 21:40:47.027235 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.027213    2575 generic.go:356] "Generic (PLEG): container finished" podID="84495d20-9ad8-4b52-b8d6-777d4f055ee2" containerID="13ac18e569bb41aa086b90cd7aafb1835746c518ef631a9582d39aca6f55864d" exitCode=0
Sep 15 21:40:47.027337 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.027265    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"
Sep 15 21:40:47.027337 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.027304    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx" event={"ID":"84495d20-9ad8-4b52-b8d6-777d4f055ee2","Type":"ContainerDied","Data":"13ac18e569bb41aa086b90cd7aafb1835746c518ef631a9582d39aca6f55864d"}
Sep 15 21:40:47.027416 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.027386    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx" event={"ID":"84495d20-9ad8-4b52-b8d6-777d4f055ee2","Type":"ContainerDied","Data":"dd1555241d9d9ce3ce4df43f2a1cbdb3967cc8328324ade2ff10403b8fa98897"}
Sep 15 21:40:47.037228 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.037208    2575 scope.go:117] "RemoveContainer" containerID="381b7ab47d4533ee1b031c5c842a392fe661f91010e15b0c9182f004fb81b986"
Sep 15 21:40:47.037552 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:40:47.037532    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"381b7ab47d4533ee1b031c5c842a392fe661f91010e15b0c9182f004fb81b986\": container with ID starting with 381b7ab47d4533ee1b031c5c842a392fe661f91010e15b0c9182f004fb81b986 not found: ID does not exist" containerID="381b7ab47d4533ee1b031c5c842a392fe661f91010e15b0c9182f004fb81b986"
Sep 15 21:40:47.037625 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.037560    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"381b7ab47d4533ee1b031c5c842a392fe661f91010e15b0c9182f004fb81b986"} err="failed to get container status \"381b7ab47d4533ee1b031c5c842a392fe661f91010e15b0c9182f004fb81b986\": rpc error: code = NotFound desc = could not find container \"381b7ab47d4533ee1b031c5c842a392fe661f91010e15b0c9182f004fb81b986\": container with ID starting with 381b7ab47d4533ee1b031c5c842a392fe661f91010e15b0c9182f004fb81b986 not found: ID does not exist"
Sep 15 21:40:47.037625 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.037576    2575 scope.go:117] "RemoveContainer" containerID="13ac18e569bb41aa086b90cd7aafb1835746c518ef631a9582d39aca6f55864d"
Sep 15 21:40:47.046993 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.046968    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d107b42-3da1-4437-b321-0669ce80dd00-plugins-config-volume\") pod \"7d107b42-3da1-4437-b321-0669ce80dd00\" (UID: \"7d107b42-3da1-4437-b321-0669ce80dd00\") "
Sep 15 21:40:47.047110 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.047036    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcs8c\" (UniqueName: \"kubernetes.io/projected/7d107b42-3da1-4437-b321-0669ce80dd00-kube-api-access-dcs8c\") pod \"7d107b42-3da1-4437-b321-0669ce80dd00\" (UID: \"7d107b42-3da1-4437-b321-0669ce80dd00\") "
Sep 15 21:40:47.047177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.047146    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w57p6\" (UniqueName: \"kubernetes.io/projected/84495d20-9ad8-4b52-b8d6-777d4f055ee2-kube-api-access-w57p6\") pod \"84495d20-9ad8-4b52-b8d6-777d4f055ee2\" (UID: \"84495d20-9ad8-4b52-b8d6-777d4f055ee2\") "
Sep 15 21:40:47.047316 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.047299    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84495d20-9ad8-4b52-b8d6-777d4f055ee2-plugins-config-volume\") pod \"84495d20-9ad8-4b52-b8d6-777d4f055ee2\" (UID: \"84495d20-9ad8-4b52-b8d6-777d4f055ee2\") "
Sep 15 21:40:47.047581 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.047385    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d107b42-3da1-4437-b321-0669ce80dd00-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d107b42-3da1-4437-b321-0669ce80dd00" (UID: "7d107b42-3da1-4437-b321-0669ce80dd00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:40:47.047581 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.047471    2575 scope.go:117] "RemoveContainer" containerID="13ac18e569bb41aa086b90cd7aafb1835746c518ef631a9582d39aca6f55864d"
Sep 15 21:40:47.047718 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.047650    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d107b42-3da1-4437-b321-0669ce80dd00-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:47.047924 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:40:47.047904    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ac18e569bb41aa086b90cd7aafb1835746c518ef631a9582d39aca6f55864d\": container with ID starting with 13ac18e569bb41aa086b90cd7aafb1835746c518ef631a9582d39aca6f55864d not found: ID does not exist" containerID="13ac18e569bb41aa086b90cd7aafb1835746c518ef631a9582d39aca6f55864d"
Sep 15 21:40:47.047999 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.047931    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84495d20-9ad8-4b52-b8d6-777d4f055ee2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84495d20-9ad8-4b52-b8d6-777d4f055ee2" (UID: "84495d20-9ad8-4b52-b8d6-777d4f055ee2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:40:47.047999 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.047934    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ac18e569bb41aa086b90cd7aafb1835746c518ef631a9582d39aca6f55864d"} err="failed to get container status \"13ac18e569bb41aa086b90cd7aafb1835746c518ef631a9582d39aca6f55864d\": rpc error: code = NotFound desc = could not find container \"13ac18e569bb41aa086b90cd7aafb1835746c518ef631a9582d39aca6f55864d\": container with ID starting with 13ac18e569bb41aa086b90cd7aafb1835746c518ef631a9582d39aca6f55864d not found: ID does not exist"
Sep 15 21:40:47.049539 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.049521    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84495d20-9ad8-4b52-b8d6-777d4f055ee2-kube-api-access-w57p6" (OuterVolumeSpecName: "kube-api-access-w57p6") pod "84495d20-9ad8-4b52-b8d6-777d4f055ee2" (UID: "84495d20-9ad8-4b52-b8d6-777d4f055ee2"). InnerVolumeSpecName "kube-api-access-w57p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:40:47.049640 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.049623    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d107b42-3da1-4437-b321-0669ce80dd00-kube-api-access-dcs8c" (OuterVolumeSpecName: "kube-api-access-dcs8c") pod "7d107b42-3da1-4437-b321-0669ce80dd00" (UID: "7d107b42-3da1-4437-b321-0669ce80dd00"). InnerVolumeSpecName "kube-api-access-dcs8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:40:47.149058 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.149029    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84495d20-9ad8-4b52-b8d6-777d4f055ee2-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:47.149058 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.149054    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcs8c\" (UniqueName: \"kubernetes.io/projected/7d107b42-3da1-4437-b321-0669ce80dd00-kube-api-access-dcs8c\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:47.149058 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.149066    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w57p6\" (UniqueName: \"kubernetes.io/projected/84495d20-9ad8-4b52-b8d6-777d4f055ee2-kube-api-access-w57p6\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:47.350253 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.350223    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"]
Sep 15 21:40:47.353606 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.353577    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ee52fdef-74dcfc8c6b-cgnmx"]
Sep 15 21:40:47.363589 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.363544    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"]
Sep 15 21:40:47.374270 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.374236    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ee52fdef-54ffbcf598-p555w"]
Sep 15 21:40:47.740031 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.740000    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"]
Sep 15 21:40:47.740633 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.740612    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84495d20-9ad8-4b52-b8d6-777d4f055ee2" containerName="payload-pre-processing"
Sep 15 21:40:47.740732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.740635    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="84495d20-9ad8-4b52-b8d6-777d4f055ee2" containerName="payload-pre-processing"
Sep 15 21:40:47.740732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.740651    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c10ef0e9-e3c4-419f-80a1-2e5a973c7950" containerName="revoke-keys"
Sep 15 21:40:47.740732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.740658    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c10ef0e9-e3c4-419f-80a1-2e5a973c7950" containerName="revoke-keys"
Sep 15 21:40:47.740732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.740687    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d107b42-3da1-4437-b321-0669ce80dd00" containerName="payload-processing"
Sep 15 21:40:47.740732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.740696    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d107b42-3da1-4437-b321-0669ce80dd00" containerName="payload-processing"
Sep 15 21:40:47.740998 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.740815    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c10ef0e9-e3c4-419f-80a1-2e5a973c7950" containerName="revoke-keys"
Sep 15 21:40:47.740998 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.740831    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="84495d20-9ad8-4b52-b8d6-777d4f055ee2" containerName="payload-pre-processing"
Sep 15 21:40:47.740998 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.740845    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d107b42-3da1-4437-b321-0669ce80dd00" containerName="payload-processing"
Sep 15 21:40:47.743828 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.743806    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.746011 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.745992    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8316f91a-openshift-default-dockercfg-ltznq\""
Sep 15 21:40:47.756373 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.756353    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"]
Sep 15 21:40:47.855269 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.855238    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-credential-socket\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.855446 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.855276    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-data\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.855446 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.855313    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-workload-socket\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.855446 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.855338    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4426644-b394-4272-b4d3-af22fd8f03d6-istiod-ca-cert\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.855446 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.855430    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-podinfo\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.855446 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.855445    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-token\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.855663 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.855462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-workload-certs\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.855663 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.855570    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tppkr\" (UniqueName: \"kubernetes.io/projected/c4426644-b394-4272-b4d3-af22fd8f03d6-kube-api-access-tppkr\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.855663 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.855613    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-envoy\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.956454 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.956360    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-workload-socket\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.956454 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.956397    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4426644-b394-4272-b4d3-af22fd8f03d6-istiod-ca-cert\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.956454 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.956451    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-podinfo\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.956736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.956468    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-token\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.956736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.956484    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-workload-certs\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.956736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.956523    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tppkr\" (UniqueName: \"kubernetes.io/projected/c4426644-b394-4272-b4d3-af22fd8f03d6-kube-api-access-tppkr\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.956736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.956562    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-envoy\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.956736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.956607    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-credential-socket\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.956736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.956637    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-data\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.957079 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.956884    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-workload-socket\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.957079 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.956924    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-workload-certs\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.957079 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.957043    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-data\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.957192 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.957098    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-credential-socket\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.957374 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.957358    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4426644-b394-4272-b4d3-af22fd8f03d6-istiod-ca-cert\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.959027 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.959006    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-envoy\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.959195 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.959175    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-podinfo\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.963951 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.963932    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-token\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:47.964105 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:47.964087    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tppkr\" (UniqueName: \"kubernetes.io/projected/c4426644-b394-4272-b4d3-af22fd8f03d6-kube-api-access-tppkr\") pod \"e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") " pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:48.054597 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:48.054560    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:48.329528 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:48.329505    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"]
Sep 15 21:40:48.343785 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:48.343723    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:40:48.343892 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:48.343805    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:40:48.343892 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:48.343831    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:40:48.747961 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:48.747928    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d107b42-3da1-4437-b321-0669ce80dd00" path="/var/lib/kubelet/pods/7d107b42-3da1-4437-b321-0669ce80dd00/volumes"
Sep 15 21:40:48.748362 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:48.748303    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84495d20-9ad8-4b52-b8d6-777d4f055ee2" path="/var/lib/kubelet/pods/84495d20-9ad8-4b52-b8d6-777d4f055ee2/volumes"
Sep 15 21:40:48.748586 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:48.748573    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c10ef0e9-e3c4-419f-80a1-2e5a973c7950" path="/var/lib/kubelet/pods/c10ef0e9-e3c4-419f-80a1-2e5a973c7950/volumes"
Sep 15 21:40:49.042284 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:49.042183    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc" event={"ID":"c4426644-b394-4272-b4d3-af22fd8f03d6","Type":"ContainerStarted","Data":"9f25257d3e45e5f0700ea02c32fcc60fc52b37197afd32958c797c2cdc958274"}
Sep 15 21:40:49.042284 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:49.042235    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc" event={"ID":"c4426644-b394-4272-b4d3-af22fd8f03d6","Type":"ContainerStarted","Data":"5676de0def1478faf5a49f8ef6310e8fb1cbe0db3b4b99d2c15e232af9e4d19b"}
Sep 15 21:40:49.055336 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:49.055304    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:49.061273 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:49.060888    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:49.062596 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:49.062551    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc" podStartSLOduration=2.062536933 podStartE2EDuration="2.062536933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:40:49.060194529 +0000 UTC m=+886.872202974" watchObservedRunningTime="2026-09-15 21:40:49.062536933 +0000 UTC m=+886.874545376"
Sep 15 21:40:50.047990 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:50.047958    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:50.049067 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:50.049051    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:40:51.643670 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:51.643634    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:40:52.247393 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.247368    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:52.303218 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.303136    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af89c3ce-6dda-46ea-8962-424a22d9ba36-istiod-ca-cert\") pod \"af89c3ce-6dda-46ea-8962-424a22d9ba36\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") "
Sep 15 21:40:52.303218 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.303176    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-workload-socket\") pod \"af89c3ce-6dda-46ea-8962-424a22d9ba36\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") "
Sep 15 21:40:52.303218 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.303196    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-token\") pod \"af89c3ce-6dda-46ea-8962-424a22d9ba36\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") "
Sep 15 21:40:52.303483 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.303246    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-podinfo\") pod \"af89c3ce-6dda-46ea-8962-424a22d9ba36\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") "
Sep 15 21:40:52.303483 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.303285    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-workload-certs\") pod \"af89c3ce-6dda-46ea-8962-424a22d9ba36\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") "
Sep 15 21:40:52.303483 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.303368    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-credential-socket\") pod \"af89c3ce-6dda-46ea-8962-424a22d9ba36\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") "
Sep 15 21:40:52.303483 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.303420    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krb85\" (UniqueName: \"kubernetes.io/projected/af89c3ce-6dda-46ea-8962-424a22d9ba36-kube-api-access-krb85\") pod \"af89c3ce-6dda-46ea-8962-424a22d9ba36\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") "
Sep 15 21:40:52.303483 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.303444    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-envoy\") pod \"af89c3ce-6dda-46ea-8962-424a22d9ba36\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") "
Sep 15 21:40:52.303483 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.303472    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af89c3ce-6dda-46ea-8962-424a22d9ba36" (UID: "af89c3ce-6dda-46ea-8962-424a22d9ba36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:52.303813 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.303527    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-data\") pod \"af89c3ce-6dda-46ea-8962-424a22d9ba36\" (UID: \"af89c3ce-6dda-46ea-8962-424a22d9ba36\") "
Sep 15 21:40:52.303813 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.303621    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af89c3ce-6dda-46ea-8962-424a22d9ba36-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af89c3ce-6dda-46ea-8962-424a22d9ba36" (UID: "af89c3ce-6dda-46ea-8962-424a22d9ba36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:40:52.303920 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.303832    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af89c3ce-6dda-46ea-8962-424a22d9ba36" (UID: "af89c3ce-6dda-46ea-8962-424a22d9ba36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:52.303920 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.303849    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-data" (OuterVolumeSpecName: "istio-data") pod "af89c3ce-6dda-46ea-8962-424a22d9ba36" (UID: "af89c3ce-6dda-46ea-8962-424a22d9ba36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:52.303920 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.303887    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af89c3ce-6dda-46ea-8962-424a22d9ba36" (UID: "af89c3ce-6dda-46ea-8962-424a22d9ba36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:52.304242 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.304082    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af89c3ce-6dda-46ea-8962-424a22d9ba36-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:52.304242 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.304101    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:52.304242 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.304115    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:52.304242 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.304129    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:52.304242 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.304142    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:52.305739 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.305715    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af89c3ce-6dda-46ea-8962-424a22d9ba36" (UID: "af89c3ce-6dda-46ea-8962-424a22d9ba36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:40:52.305739 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.305720    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-token" (OuterVolumeSpecName: "istio-token") pod "af89c3ce-6dda-46ea-8962-424a22d9ba36" (UID: "af89c3ce-6dda-46ea-8962-424a22d9ba36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:40:52.305960 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.305744    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af89c3ce-6dda-46ea-8962-424a22d9ba36-kube-api-access-krb85" (OuterVolumeSpecName: "kube-api-access-krb85") pod "af89c3ce-6dda-46ea-8962-424a22d9ba36" (UID: "af89c3ce-6dda-46ea-8962-424a22d9ba36"). InnerVolumeSpecName "kube-api-access-krb85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:40:52.305960 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.305884    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af89c3ce-6dda-46ea-8962-424a22d9ba36" (UID: "af89c3ce-6dda-46ea-8962-424a22d9ba36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:40:52.404716 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.404679    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krb85\" (UniqueName: \"kubernetes.io/projected/af89c3ce-6dda-46ea-8962-424a22d9ba36-kube-api-access-krb85\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:52.404716 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.404714    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:52.404716 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.404727    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:52.404984 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.404736    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af89c3ce-6dda-46ea-8962-424a22d9ba36-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:40:52.768682 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.768645    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"]
Sep 15 21:40:52.769106 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.769091    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af89c3ce-6dda-46ea-8962-424a22d9ba36" containerName="istio-proxy"
Sep 15 21:40:52.769106 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.769107    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="af89c3ce-6dda-46ea-8962-424a22d9ba36" containerName="istio-proxy"
Sep 15 21:40:52.769209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.769120    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af89c3ce-6dda-46ea-8962-424a22d9ba36" containerName="istio-proxy"
Sep 15 21:40:52.769209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.769128    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="af89c3ce-6dda-46ea-8962-424a22d9ba36" containerName="istio-proxy"
Sep 15 21:40:52.769209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.769198    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="af89c3ce-6dda-46ea-8962-424a22d9ba36" containerName="istio-proxy"
Sep 15 21:40:52.771429 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.771412    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"
Sep 15 21:40:52.774067 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.774047    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8316f91a-dockercfg-tq9zt\""
Sep 15 21:40:52.774188 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.774050    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8316f91a\""
Sep 15 21:40:52.783912 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.783886    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"]
Sep 15 21:40:52.831855 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.831822    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"]
Sep 15 21:40:52.832353 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.832340    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="af89c3ce-6dda-46ea-8962-424a22d9ba36" containerName="istio-proxy"
Sep 15 21:40:52.834534 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.834518    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"
Sep 15 21:40:52.844249 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.844225    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"]
Sep 15 21:40:52.911118 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.911081    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94c19b7-7c53-42e5-8ea7-709c2175ab6d-plugins-config-volume\") pod \"payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc\" (UID: \"c94c19b7-7c53-42e5-8ea7-709c2175ab6d\") " pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"
Sep 15 21:40:52.911274 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.911131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tk2m\" (UniqueName: \"kubernetes.io/projected/9a6687d5-80b4-4e49-aa50-6903cc114efb-kube-api-access-8tk2m\") pod \"payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd\" (UID: \"9a6687d5-80b4-4e49-aa50-6903cc114efb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"
Sep 15 21:40:52.911274 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.911192    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6687d5-80b4-4e49-aa50-6903cc114efb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd\" (UID: \"9a6687d5-80b4-4e49-aa50-6903cc114efb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"
Sep 15 21:40:52.911274 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:52.911269    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b5mc\" (UniqueName: \"kubernetes.io/projected/c94c19b7-7c53-42e5-8ea7-709c2175ab6d-kube-api-access-4b5mc\") pod \"payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc\" (UID: \"c94c19b7-7c53-42e5-8ea7-709c2175ab6d\") " pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"
Sep 15 21:40:53.012234 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.012120    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b5mc\" (UniqueName: \"kubernetes.io/projected/c94c19b7-7c53-42e5-8ea7-709c2175ab6d-kube-api-access-4b5mc\") pod \"payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc\" (UID: \"c94c19b7-7c53-42e5-8ea7-709c2175ab6d\") " pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"
Sep 15 21:40:53.012418 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.012244    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94c19b7-7c53-42e5-8ea7-709c2175ab6d-plugins-config-volume\") pod \"payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc\" (UID: \"c94c19b7-7c53-42e5-8ea7-709c2175ab6d\") " pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"
Sep 15 21:40:53.012418 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.012295    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tk2m\" (UniqueName: \"kubernetes.io/projected/9a6687d5-80b4-4e49-aa50-6903cc114efb-kube-api-access-8tk2m\") pod \"payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd\" (UID: \"9a6687d5-80b4-4e49-aa50-6903cc114efb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"
Sep 15 21:40:53.012418 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.012332    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6687d5-80b4-4e49-aa50-6903cc114efb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd\" (UID: \"9a6687d5-80b4-4e49-aa50-6903cc114efb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"
Sep 15 21:40:53.013069 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.013040    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94c19b7-7c53-42e5-8ea7-709c2175ab6d-plugins-config-volume\") pod \"payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc\" (UID: \"c94c19b7-7c53-42e5-8ea7-709c2175ab6d\") " pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"
Sep 15 21:40:53.013160 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.013047    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6687d5-80b4-4e49-aa50-6903cc114efb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd\" (UID: \"9a6687d5-80b4-4e49-aa50-6903cc114efb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"
Sep 15 21:40:53.023561 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.023539    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tk2m\" (UniqueName: \"kubernetes.io/projected/9a6687d5-80b4-4e49-aa50-6903cc114efb-kube-api-access-8tk2m\") pod \"payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd\" (UID: \"9a6687d5-80b4-4e49-aa50-6903cc114efb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"
Sep 15 21:40:53.023678 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.023573    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b5mc\" (UniqueName: \"kubernetes.io/projected/c94c19b7-7c53-42e5-8ea7-709c2175ab6d-kube-api-access-4b5mc\") pod \"payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc\" (UID: \"c94c19b7-7c53-42e5-8ea7-709c2175ab6d\") " pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"
Sep 15 21:40:53.066139 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.066096    2575 generic.go:356] "Generic (PLEG): container finished" podID="af89c3ce-6dda-46ea-8962-424a22d9ba36" containerID="e28abe7f2ea838689ae7941afb7406343fc38046b5646bacb3bd728046dd13b7" exitCode=0
Sep 15 21:40:53.066321 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.066145    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl" event={"ID":"af89c3ce-6dda-46ea-8962-424a22d9ba36","Type":"ContainerDied","Data":"e28abe7f2ea838689ae7941afb7406343fc38046b5646bacb3bd728046dd13b7"}
Sep 15 21:40:53.066321 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.066228    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl" event={"ID":"af89c3ce-6dda-46ea-8962-424a22d9ba36","Type":"ContainerDied","Data":"bba42cd897e8b19be1237ab2027c47d1eb458493ef3e350dee445e6ef6f6c2f1"}
Sep 15 21:40:53.066321 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.066157    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"
Sep 15 21:40:53.066321 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.066245    2575 scope.go:117] "RemoveContainer" containerID="e28abe7f2ea838689ae7941afb7406343fc38046b5646bacb3bd728046dd13b7"
Sep 15 21:40:53.076079 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.076059    2575 scope.go:117] "RemoveContainer" containerID="bae3cb1d52e700be0b691d94227181dfb70dfa25d6dec3970e28bf417de87293"
Sep 15 21:40:53.082642 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.082614    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"]
Sep 15 21:40:53.082803 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.082716    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"
Sep 15 21:40:53.085913 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.085796    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ee52fdef-gw-openshift-default-fcb497896-5nkvl"]
Sep 15 21:40:53.085913 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.085880    2575 scope.go:117] "RemoveContainer" containerID="e28abe7f2ea838689ae7941afb7406343fc38046b5646bacb3bd728046dd13b7"
Sep 15 21:40:53.086174 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:40:53.086155    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e28abe7f2ea838689ae7941afb7406343fc38046b5646bacb3bd728046dd13b7\": container with ID starting with e28abe7f2ea838689ae7941afb7406343fc38046b5646bacb3bd728046dd13b7 not found: ID does not exist" containerID="e28abe7f2ea838689ae7941afb7406343fc38046b5646bacb3bd728046dd13b7"
Sep 15 21:40:53.086223 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.086182    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28abe7f2ea838689ae7941afb7406343fc38046b5646bacb3bd728046dd13b7"} err="failed to get container status \"e28abe7f2ea838689ae7941afb7406343fc38046b5646bacb3bd728046dd13b7\": rpc error: code = NotFound desc = could not find container \"e28abe7f2ea838689ae7941afb7406343fc38046b5646bacb3bd728046dd13b7\": container with ID starting with e28abe7f2ea838689ae7941afb7406343fc38046b5646bacb3bd728046dd13b7 not found: ID does not exist"
Sep 15 21:40:53.086223 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.086210    2575 scope.go:117] "RemoveContainer" containerID="bae3cb1d52e700be0b691d94227181dfb70dfa25d6dec3970e28bf417de87293"
Sep 15 21:40:53.086494 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:40:53.086476    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae3cb1d52e700be0b691d94227181dfb70dfa25d6dec3970e28bf417de87293\": container with ID starting with bae3cb1d52e700be0b691d94227181dfb70dfa25d6dec3970e28bf417de87293 not found: ID does not exist" containerID="bae3cb1d52e700be0b691d94227181dfb70dfa25d6dec3970e28bf417de87293"
Sep 15 21:40:53.086530 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.086500    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae3cb1d52e700be0b691d94227181dfb70dfa25d6dec3970e28bf417de87293"} err="failed to get container status \"bae3cb1d52e700be0b691d94227181dfb70dfa25d6dec3970e28bf417de87293\": rpc error: code = NotFound desc = could not find container \"bae3cb1d52e700be0b691d94227181dfb70dfa25d6dec3970e28bf417de87293\": container with ID starting with bae3cb1d52e700be0b691d94227181dfb70dfa25d6dec3970e28bf417de87293 not found: ID does not exist"
Sep 15 21:40:53.145161 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.145127    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"
Sep 15 21:40:53.359798 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.359776    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"]
Sep 15 21:40:53.361802 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:40:53.361747    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94c19b7_7c53_42e5_8ea7_709c2175ab6d.slice/crio-e39fbf573a68de453061bd6c21ac62e8113c970cc7774d3dabee5c798ffbd6cd WatchSource:0}: Error finding container e39fbf573a68de453061bd6c21ac62e8113c970cc7774d3dabee5c798ffbd6cd: Status 404 returned error can't find the container with id e39fbf573a68de453061bd6c21ac62e8113c970cc7774d3dabee5c798ffbd6cd
Sep 15 21:40:53.440185 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:53.440159    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"]
Sep 15 21:40:53.442707 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:40:53.442681    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6687d5_80b4_4e49_aa50_6903cc114efb.slice/crio-3e4194a83c39de47ecaa44836609da23e1b9838387a33ca643145c206b967809 WatchSource:0}: Error finding container 3e4194a83c39de47ecaa44836609da23e1b9838387a33ca643145c206b967809: Status 404 returned error can't find the container with id 3e4194a83c39de47ecaa44836609da23e1b9838387a33ca643145c206b967809
Sep 15 21:40:54.074839 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:54.074799    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc" event={"ID":"c94c19b7-7c53-42e5-8ea7-709c2175ab6d","Type":"ContainerStarted","Data":"670d813a8a72497314030b3267826e39671cf9afbf24b5612bd34a053d16adca"}
Sep 15 21:40:54.075292 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:54.074861    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc" event={"ID":"c94c19b7-7c53-42e5-8ea7-709c2175ab6d","Type":"ContainerStarted","Data":"e39fbf573a68de453061bd6c21ac62e8113c970cc7774d3dabee5c798ffbd6cd"}
Sep 15 21:40:54.075292 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:54.074963    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"
Sep 15 21:40:54.076471 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:54.076447    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd" event={"ID":"9a6687d5-80b4-4e49-aa50-6903cc114efb","Type":"ContainerStarted","Data":"b36a08d5db81968566a862e94fde76254573b2c6ceb210d20dc0acd3736237af"}
Sep 15 21:40:54.076586 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:54.076492    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd" event={"ID":"9a6687d5-80b4-4e49-aa50-6903cc114efb","Type":"ContainerStarted","Data":"3e4194a83c39de47ecaa44836609da23e1b9838387a33ca643145c206b967809"}
Sep 15 21:40:54.076586 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:54.076533    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"
Sep 15 21:40:54.091681 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:54.091629    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc" podStartSLOduration=2.091613668 podStartE2EDuration="2.091613668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:40:54.089100536 +0000 UTC m=+891.901108980" watchObservedRunningTime="2026-09-15 21:40:54.091613668 +0000 UTC m=+891.903622111"
Sep 15 21:40:54.106072 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:54.106023    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd" podStartSLOduration=2.106007401 podStartE2EDuration="2.106007401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:40:54.103738179 +0000 UTC m=+891.915746624" watchObservedRunningTime="2026-09-15 21:40:54.106007401 +0000 UTC m=+891.918015845"
Sep 15 21:40:54.747543 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:40:54.747508    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af89c3ce-6dda-46ea-8962-424a22d9ba36" path="/var/lib/kubelet/pods/af89c3ce-6dda-46ea-8962-424a22d9ba36/volumes"
Sep 15 21:41:05.084308 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:05.084278    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"
Sep 15 21:41:05.084661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:05.084354    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"
Sep 15 21:41:09.026773 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:09.026703    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:41:13.197600 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:13.197569    2575 generic.go:356] "Generic (PLEG): container finished" podID="c4426644-b394-4272-b4d3-af22fd8f03d6" containerID="9f25257d3e45e5f0700ea02c32fcc60fc52b37197afd32958c797c2cdc958274" exitCode=0
Sep 15 21:41:13.198170 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:13.197621    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc" event={"ID":"c4426644-b394-4272-b4d3-af22fd8f03d6","Type":"ContainerDied","Data":"9f25257d3e45e5f0700ea02c32fcc60fc52b37197afd32958c797c2cdc958274"}
Sep 15 21:41:13.198238 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:13.198177    2575 scope.go:117] "RemoveContainer" containerID="9f25257d3e45e5f0700ea02c32fcc60fc52b37197afd32958c797c2cdc958274"
Sep 15 21:41:13.199348 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:13.199318    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:41:13.199440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:13.199405    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:41:13.199480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:13.199438    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:41:14.205133 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:14.205091    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc" event={"ID":"c4426644-b394-4272-b4d3-af22fd8f03d6","Type":"ContainerStarted","Data":"6b5ced23b38d6993dc60b9abf25e4942ea9b186949f595dbdcfb88a849e9f258"}
Sep 15 21:41:15.055296 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:15.055263    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:41:15.055644 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:15.055621    2575 patch_prober.go:28] interesting pod/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 15 21:41:15.055837 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:15.055811    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc" podUID="c4426644-b394-4272-b4d3-af22fd8f03d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 15 21:41:16.055701 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:16.055656    2575 patch_prober.go:28] interesting pod/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 15 21:41:16.056212 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:16.055738    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc" podUID="c4426644-b394-4272-b4d3-af22fd8f03d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 15 21:41:16.118262 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:16.118229    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n"]
Sep 15 21:41:16.120960 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:16.120943    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n"
Sep 15 21:41:16.123072 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:16.123048    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:41:16.123184 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:16.123169    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 21:41:16.128483 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:16.128361    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n"]
Sep 15 21:41:16.158778 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:16.157264    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e3abb6-7e95-4e9c-bcdf-b7299120aa33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n\" (UID: \"47e3abb6-7e95-4e9c-bcdf-b7299120aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n"
Sep 15 21:41:16.258328 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:16.258288    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e3abb6-7e95-4e9c-bcdf-b7299120aa33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n\" (UID: \"47e3abb6-7e95-4e9c-bcdf-b7299120aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n"
Sep 15 21:41:16.259288 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:16.259258    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e3abb6-7e95-4e9c-bcdf-b7299120aa33-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n\" (UID: \"47e3abb6-7e95-4e9c-bcdf-b7299120aa33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n"
Sep 15 21:41:16.432811 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:16.432742    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n"
Sep 15 21:41:16.725648 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:16.725619    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n"]
Sep 15 21:41:16.728270 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:41:16.728240    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e3abb6_7e95_4e9c_bcdf_b7299120aa33.slice/crio-49a6ee49113eeaa82df765c18380e792cdf9a40ff7c23dc854a92c2a6126066e WatchSource:0}: Error finding container 49a6ee49113eeaa82df765c18380e792cdf9a40ff7c23dc854a92c2a6126066e: Status 404 returned error can't find the container with id 49a6ee49113eeaa82df765c18380e792cdf9a40ff7c23dc854a92c2a6126066e
Sep 15 21:41:17.055809 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.055342    2575 patch_prober.go:28] interesting pod/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 15 21:41:17.055809 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.055416    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc" podUID="c4426644-b394-4272-b4d3-af22fd8f03d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 15 21:41:17.121400 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.121355    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"]
Sep 15 21:41:17.124551 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.124526    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.133840 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.133805    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"]
Sep 15 21:41:17.134053 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.134033    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-56e305\"/\"openshift-service-ca.crt\""
Sep 15 21:41:17.134327 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.134304    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-56e305\"/\"simulator-w4-56e305-kserve-self-signed-certs\""
Sep 15 21:41:17.134523 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.134505    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-56e305\"/\"default-dockercfg-tfg85\""
Sep 15 21:41:17.134698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.134680    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-56e305\"/\"kube-root-ca.crt\""
Sep 15 21:41:17.176734 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.176326    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-dshm\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.176734 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.176384    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-model-cache\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.176734 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.176436    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfhjt\" (UniqueName: \"kubernetes.io/projected/bb08497b-950e-4db1-ac06-0ba91361da32-kube-api-access-nfhjt\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.176734 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.176519    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-home\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.176734 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.176563    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb08497b-950e-4db1-ac06-0ba91361da32-tls-certs\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.176734 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.176594    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-tmp-dir\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.231867 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.231827    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n" event={"ID":"47e3abb6-7e95-4e9c-bcdf-b7299120aa33","Type":"ContainerStarted","Data":"4eda1a2ca5f833412830df5d4e7afacd813dc3ca846b23533ac36cfae99de7bc"}
Sep 15 21:41:17.232160 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.232138    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n" event={"ID":"47e3abb6-7e95-4e9c-bcdf-b7299120aa33","Type":"ContainerStarted","Data":"49a6ee49113eeaa82df765c18380e792cdf9a40ff7c23dc854a92c2a6126066e"}
Sep 15 21:41:17.247912 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.247867    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n" podStartSLOduration=1.247854541 podStartE2EDuration="1.247854541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:17.246717071 +0000 UTC m=+915.058725514" watchObservedRunningTime="2026-09-15 21:41:17.247854541 +0000 UTC m=+915.059862984"
Sep 15 21:41:17.277402 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.277360    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb08497b-950e-4db1-ac06-0ba91361da32-tls-certs\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.277600 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.277407    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-tmp-dir\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.277600 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.277484    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-dshm\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.277600 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.277514    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-model-cache\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.277600 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.277561    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfhjt\" (UniqueName: \"kubernetes.io/projected/bb08497b-950e-4db1-ac06-0ba91361da32-kube-api-access-nfhjt\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.277869 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.277648    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-home\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.277933 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.277877    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-tmp-dir\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.278164 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.278124    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-model-cache\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.278264 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.278218    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-home\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.280054 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.280030    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-dshm\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.280246 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.280229    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb08497b-950e-4db1-ac06-0ba91361da32-tls-certs\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.285147 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.285126    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfhjt\" (UniqueName: \"kubernetes.io/projected/bb08497b-950e-4db1-ac06-0ba91361da32-kube-api-access-nfhjt\") pod \"simulator-w4-56e305-kserve-798766b64f-z8g4d\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") " pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.445418 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.445379    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:17.744368 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:17.744344    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"]
Sep 15 21:41:17.746260 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:41:17.746233    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb08497b_950e_4db1_ac06_0ba91361da32.slice/crio-118df60534f4d9810c0c8eb75db2449d27c0b47359e8d487aaad2d3d0c3782fa WatchSource:0}: Error finding container 118df60534f4d9810c0c8eb75db2449d27c0b47359e8d487aaad2d3d0c3782fa: Status 404 returned error can't find the container with id 118df60534f4d9810c0c8eb75db2449d27c0b47359e8d487aaad2d3d0c3782fa
Sep 15 21:41:18.055386 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:18.055289    2575 patch_prober.go:28] interesting pod/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 15 21:41:18.056078 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:18.055643    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc" podUID="c4426644-b394-4272-b4d3-af22fd8f03d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 15 21:41:18.242464 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:18.242425    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d" event={"ID":"bb08497b-950e-4db1-ac06-0ba91361da32","Type":"ContainerStarted","Data":"330c208356462a6419547833ce82348c17a196600a47eeb0a488efd40e81dc3e"}
Sep 15 21:41:18.242464 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:18.242473    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d" event={"ID":"bb08497b-950e-4db1-ac06-0ba91361da32","Type":"ContainerStarted","Data":"118df60534f4d9810c0c8eb75db2449d27c0b47359e8d487aaad2d3d0c3782fa"}
Sep 15 21:41:18.244047 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:18.244024    2575 generic.go:356] "Generic (PLEG): container finished" podID="47e3abb6-7e95-4e9c-bcdf-b7299120aa33" containerID="4eda1a2ca5f833412830df5d4e7afacd813dc3ca846b23533ac36cfae99de7bc" exitCode=0
Sep 15 21:41:18.244151 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:18.244083    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n" event={"ID":"47e3abb6-7e95-4e9c-bcdf-b7299120aa33","Type":"ContainerDied","Data":"4eda1a2ca5f833412830df5d4e7afacd813dc3ca846b23533ac36cfae99de7bc"}
Sep 15 21:41:18.280165 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:18.280103    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d" podStartSLOduration=1.2800842700000001 podStartE2EDuration="1.28008427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:18.278041721 +0000 UTC m=+916.090050188" watchObservedRunningTime="2026-09-15 21:41:18.28008427 +0000 UTC m=+916.092092714"
Sep 15 21:41:19.055446 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:19.055402    2575 patch_prober.go:28] interesting pod/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 15 21:41:19.055657 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:19.055481    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc" podUID="c4426644-b394-4272-b4d3-af22fd8f03d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 15 21:41:19.399955 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:19.399925    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n"
Sep 15 21:41:19.504687 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:19.504657    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e3abb6-7e95-4e9c-bcdf-b7299120aa33-maas-api-service-ca\") pod \"47e3abb6-7e95-4e9c-bcdf-b7299120aa33\" (UID: \"47e3abb6-7e95-4e9c-bcdf-b7299120aa33\") "
Sep 15 21:41:19.505019 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:19.504993    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e3abb6-7e95-4e9c-bcdf-b7299120aa33-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47e3abb6-7e95-4e9c-bcdf-b7299120aa33" (UID: "47e3abb6-7e95-4e9c-bcdf-b7299120aa33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:41:19.606136 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:19.606111    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47e3abb6-7e95-4e9c-bcdf-b7299120aa33-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:41:20.048480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:20.048385    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:41:20.059100 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:20.059076    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:41:20.256184 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:20.256151    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n"
Sep 15 21:41:20.256371 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:20.256193    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n" event={"ID":"47e3abb6-7e95-4e9c-bcdf-b7299120aa33","Type":"ContainerDied","Data":"49a6ee49113eeaa82df765c18380e792cdf9a40ff7c23dc854a92c2a6126066e"}
Sep 15 21:41:20.256371 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:20.256234    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="49a6ee49113eeaa82df765c18380e792cdf9a40ff7c23dc854a92c2a6126066e"
Sep 15 21:41:20.257437 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:20.257416    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:41:21.722788 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:21.722736    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"]
Sep 15 21:41:21.723207 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:21.723019    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc" podUID="c94c19b7-7c53-42e5-8ea7-709c2175ab6d" containerName="payload-processing" containerID="cri-o://670d813a8a72497314030b3267826e39671cf9afbf24b5612bd34a053d16adca" gracePeriod=30
Sep 15 21:41:21.734202 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:21.732856    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"]
Sep 15 21:41:21.734202 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:21.733247    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd" podUID="9a6687d5-80b4-4e49-aa50-6903cc114efb" containerName="payload-pre-processing" containerID="cri-o://b36a08d5db81968566a862e94fde76254573b2c6ceb210d20dc0acd3736237af" gracePeriod=30
Sep 15 21:41:21.874193 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:21.874145    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n"]
Sep 15 21:41:21.876998 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:21.876970    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8316f91a-qgh6n"]
Sep 15 21:41:22.007537 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.007498    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"
Sep 15 21:41:22.037627 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.037591    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"
Sep 15 21:41:22.130287 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.130248    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94c19b7-7c53-42e5-8ea7-709c2175ab6d-plugins-config-volume\") pod \"c94c19b7-7c53-42e5-8ea7-709c2175ab6d\" (UID: \"c94c19b7-7c53-42e5-8ea7-709c2175ab6d\") "
Sep 15 21:41:22.130287 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.130304    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b5mc\" (UniqueName: \"kubernetes.io/projected/c94c19b7-7c53-42e5-8ea7-709c2175ab6d-kube-api-access-4b5mc\") pod \"c94c19b7-7c53-42e5-8ea7-709c2175ab6d\" (UID: \"c94c19b7-7c53-42e5-8ea7-709c2175ab6d\") "
Sep 15 21:41:22.130527 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.130349    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6687d5-80b4-4e49-aa50-6903cc114efb-plugins-config-volume\") pod \"9a6687d5-80b4-4e49-aa50-6903cc114efb\" (UID: \"9a6687d5-80b4-4e49-aa50-6903cc114efb\") "
Sep 15 21:41:22.130527 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.130368    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tk2m\" (UniqueName: \"kubernetes.io/projected/9a6687d5-80b4-4e49-aa50-6903cc114efb-kube-api-access-8tk2m\") pod \"9a6687d5-80b4-4e49-aa50-6903cc114efb\" (UID: \"9a6687d5-80b4-4e49-aa50-6903cc114efb\") "
Sep 15 21:41:22.130713 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.130688    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94c19b7-7c53-42e5-8ea7-709c2175ab6d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c94c19b7-7c53-42e5-8ea7-709c2175ab6d" (UID: "c94c19b7-7c53-42e5-8ea7-709c2175ab6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:41:22.130796 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.130780    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a6687d5-80b4-4e49-aa50-6903cc114efb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9a6687d5-80b4-4e49-aa50-6903cc114efb" (UID: "9a6687d5-80b4-4e49-aa50-6903cc114efb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:41:22.132666 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.132644    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a6687d5-80b4-4e49-aa50-6903cc114efb-kube-api-access-8tk2m" (OuterVolumeSpecName: "kube-api-access-8tk2m") pod "9a6687d5-80b4-4e49-aa50-6903cc114efb" (UID: "9a6687d5-80b4-4e49-aa50-6903cc114efb"). InnerVolumeSpecName "kube-api-access-8tk2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:41:22.132738 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.132681    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94c19b7-7c53-42e5-8ea7-709c2175ab6d-kube-api-access-4b5mc" (OuterVolumeSpecName: "kube-api-access-4b5mc") pod "c94c19b7-7c53-42e5-8ea7-709c2175ab6d" (UID: "c94c19b7-7c53-42e5-8ea7-709c2175ab6d"). InnerVolumeSpecName "kube-api-access-4b5mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:41:22.231460 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.231430    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a6687d5-80b4-4e49-aa50-6903cc114efb-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:41:22.231460 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.231460    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tk2m\" (UniqueName: \"kubernetes.io/projected/9a6687d5-80b4-4e49-aa50-6903cc114efb-kube-api-access-8tk2m\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:41:22.231658 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.231473    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c94c19b7-7c53-42e5-8ea7-709c2175ab6d-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:41:22.231658 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.231486    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b5mc\" (UniqueName: \"kubernetes.io/projected/c94c19b7-7c53-42e5-8ea7-709c2175ab6d-kube-api-access-4b5mc\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:41:22.268468 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.268369    2575 generic.go:356] "Generic (PLEG): container finished" podID="9a6687d5-80b4-4e49-aa50-6903cc114efb" containerID="b36a08d5db81968566a862e94fde76254573b2c6ceb210d20dc0acd3736237af" exitCode=0
Sep 15 21:41:22.268468 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.268415    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"
Sep 15 21:41:22.268468 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.268457    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd" event={"ID":"9a6687d5-80b4-4e49-aa50-6903cc114efb","Type":"ContainerDied","Data":"b36a08d5db81968566a862e94fde76254573b2c6ceb210d20dc0acd3736237af"}
Sep 15 21:41:22.268725 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.268509    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd" event={"ID":"9a6687d5-80b4-4e49-aa50-6903cc114efb","Type":"ContainerDied","Data":"3e4194a83c39de47ecaa44836609da23e1b9838387a33ca643145c206b967809"}
Sep 15 21:41:22.268725 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.268533    2575 scope.go:117] "RemoveContainer" containerID="b36a08d5db81968566a862e94fde76254573b2c6ceb210d20dc0acd3736237af"
Sep 15 21:41:22.269970 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.269946    2575 generic.go:356] "Generic (PLEG): container finished" podID="c94c19b7-7c53-42e5-8ea7-709c2175ab6d" containerID="670d813a8a72497314030b3267826e39671cf9afbf24b5612bd34a053d16adca" exitCode=0
Sep 15 21:41:22.270080 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.269981    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"
Sep 15 21:41:22.270080 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.269989    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc" event={"ID":"c94c19b7-7c53-42e5-8ea7-709c2175ab6d","Type":"ContainerDied","Data":"670d813a8a72497314030b3267826e39671cf9afbf24b5612bd34a053d16adca"}
Sep 15 21:41:22.270080 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.270032    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc" event={"ID":"c94c19b7-7c53-42e5-8ea7-709c2175ab6d","Type":"ContainerDied","Data":"e39fbf573a68de453061bd6c21ac62e8113c970cc7774d3dabee5c798ffbd6cd"}
Sep 15 21:41:22.277958 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.277877    2575 scope.go:117] "RemoveContainer" containerID="b36a08d5db81968566a862e94fde76254573b2c6ceb210d20dc0acd3736237af"
Sep 15 21:41:22.278194 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:41:22.278175    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36a08d5db81968566a862e94fde76254573b2c6ceb210d20dc0acd3736237af\": container with ID starting with b36a08d5db81968566a862e94fde76254573b2c6ceb210d20dc0acd3736237af not found: ID does not exist" containerID="b36a08d5db81968566a862e94fde76254573b2c6ceb210d20dc0acd3736237af"
Sep 15 21:41:22.278249 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.278207    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36a08d5db81968566a862e94fde76254573b2c6ceb210d20dc0acd3736237af"} err="failed to get container status \"b36a08d5db81968566a862e94fde76254573b2c6ceb210d20dc0acd3736237af\": rpc error: code = NotFound desc = could not find container \"b36a08d5db81968566a862e94fde76254573b2c6ceb210d20dc0acd3736237af\": container with ID starting with b36a08d5db81968566a862e94fde76254573b2c6ceb210d20dc0acd3736237af not found: ID does not exist"
Sep 15 21:41:22.278249 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.278229    2575 scope.go:117] "RemoveContainer" containerID="670d813a8a72497314030b3267826e39671cf9afbf24b5612bd34a053d16adca"
Sep 15 21:41:22.286421 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.286405    2575 scope.go:117] "RemoveContainer" containerID="670d813a8a72497314030b3267826e39671cf9afbf24b5612bd34a053d16adca"
Sep 15 21:41:22.286661 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:41:22.286644    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"670d813a8a72497314030b3267826e39671cf9afbf24b5612bd34a053d16adca\": container with ID starting with 670d813a8a72497314030b3267826e39671cf9afbf24b5612bd34a053d16adca not found: ID does not exist" containerID="670d813a8a72497314030b3267826e39671cf9afbf24b5612bd34a053d16adca"
Sep 15 21:41:22.286708 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.286667    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"670d813a8a72497314030b3267826e39671cf9afbf24b5612bd34a053d16adca"} err="failed to get container status \"670d813a8a72497314030b3267826e39671cf9afbf24b5612bd34a053d16adca\": rpc error: code = NotFound desc = could not find container \"670d813a8a72497314030b3267826e39671cf9afbf24b5612bd34a053d16adca\": container with ID starting with 670d813a8a72497314030b3267826e39671cf9afbf24b5612bd34a053d16adca not found: ID does not exist"
Sep 15 21:41:22.292494 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.292474    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"]
Sep 15 21:41:22.297931 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.297912    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8316f91a-7bdd795cd9-qdtzc"]
Sep 15 21:41:22.316658 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.316636    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"]
Sep 15 21:41:22.321873 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.321853    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8316f91a-7b88d4f5d-stkdd"]
Sep 15 21:41:22.747936 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.747905    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47e3abb6-7e95-4e9c-bcdf-b7299120aa33" path="/var/lib/kubelet/pods/47e3abb6-7e95-4e9c-bcdf-b7299120aa33/volumes"
Sep 15 21:41:22.748292 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.748275    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a6687d5-80b4-4e49-aa50-6903cc114efb" path="/var/lib/kubelet/pods/9a6687d5-80b4-4e49-aa50-6903cc114efb/volumes"
Sep 15 21:41:22.748635 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.748622    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c94c19b7-7c53-42e5-8ea7-709c2175ab6d" path="/var/lib/kubelet/pods/c94c19b7-7c53-42e5-8ea7-709c2175ab6d/volumes"
Sep 15 21:41:22.846155 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:22.846127    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:41:23.055140 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.055045    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"]
Sep 15 21:41:23.055305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.055259    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc" podUID="c4426644-b394-4272-b4d3-af22fd8f03d6" containerName="istio-proxy" containerID="cri-o://6b5ced23b38d6993dc60b9abf25e4942ea9b186949f595dbdcfb88a849e9f258" gracePeriod=30
Sep 15 21:41:23.811721 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.811692    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"]
Sep 15 21:41:23.812159 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.812145    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a6687d5-80b4-4e49-aa50-6903cc114efb" containerName="payload-pre-processing"
Sep 15 21:41:23.812199 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.812161    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a6687d5-80b4-4e49-aa50-6903cc114efb" containerName="payload-pre-processing"
Sep 15 21:41:23.812199 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.812173    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c94c19b7-7c53-42e5-8ea7-709c2175ab6d" containerName="payload-processing"
Sep 15 21:41:23.812199 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.812179    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c94c19b7-7c53-42e5-8ea7-709c2175ab6d" containerName="payload-processing"
Sep 15 21:41:23.812199 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.812195    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e3abb6-7e95-4e9c-bcdf-b7299120aa33" containerName="revoke-keys"
Sep 15 21:41:23.812322 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.812201    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e3abb6-7e95-4e9c-bcdf-b7299120aa33" containerName="revoke-keys"
Sep 15 21:41:23.812322 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.812265    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="47e3abb6-7e95-4e9c-bcdf-b7299120aa33" containerName="revoke-keys"
Sep 15 21:41:23.812322 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.812273    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c94c19b7-7c53-42e5-8ea7-709c2175ab6d" containerName="payload-processing"
Sep 15 21:41:23.812322 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.812282    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a6687d5-80b4-4e49-aa50-6903cc114efb" containerName="payload-pre-processing"
Sep 15 21:41:23.815001 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.814984    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:23.817184 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.817165    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-8ce29e8a-openshift-default-dockercfg-bmvxh\""
Sep 15 21:41:23.825645 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.825624    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"]
Sep 15 21:41:23.947969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.947937    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istiod-ca-cert\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:23.948113 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.948002    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-workload-certs\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:23.948113 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.948035    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-token\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:23.948113 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.948082    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-credential-socket\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:23.948235 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.948122    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-envoy\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:23.948235 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.948146    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-workload-socket\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:23.948235 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.948190    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-podinfo\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:23.948235 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.948227    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-data\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:23.948358 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:23.948255    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhdj2\" (UniqueName: \"kubernetes.io/projected/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-kube-api-access-vhdj2\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.049633 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.049597    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhdj2\" (UniqueName: \"kubernetes.io/projected/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-kube-api-access-vhdj2\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.049633 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.049639    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istiod-ca-cert\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.049856 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.049690    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-workload-certs\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.049856 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.049717    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-token\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.049856 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.049788    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-credential-socket\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.049856 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.049819    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-envoy\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.049856 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.049853    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-workload-socket\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.050091 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.049879    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-podinfo\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.050091 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.049907    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-data\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.050271 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.050249    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-workload-certs\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.050381 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.050360    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-workload-socket\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.050435 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.050416    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-data\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.050504 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.050458    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-credential-socket\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.050584 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.050568    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istiod-ca-cert\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.052362 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.052339    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-envoy\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.052568 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.052549    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-podinfo\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.067165 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.067111    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhdj2\" (UniqueName: \"kubernetes.io/projected/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-kube-api-access-vhdj2\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.067363 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.067344    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-token\") pod \"e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") " pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.127069 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.127040    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:24.401317 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.401290    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"]
Sep 15 21:41:24.402727 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:41:24.402704    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ca6cea_ec0c_4d55_8b9f_02c74af881e4.slice/crio-97f3adec0bd1d5ccc3f445095f4c7cea7b884dff55e7ad77790b8508d492c819 WatchSource:0}: Error finding container 97f3adec0bd1d5ccc3f445095f4c7cea7b884dff55e7ad77790b8508d492c819: Status 404 returned error can't find the container with id 97f3adec0bd1d5ccc3f445095f4c7cea7b884dff55e7ad77790b8508d492c819
Sep 15 21:41:24.404934 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.404901    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:41:24.405008 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.404965    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:41:24.405008 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:24.404992    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:41:25.289470 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:25.289427    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9" event={"ID":"03ca6cea-ec0c-4d55-8b9f-02c74af881e4","Type":"ContainerStarted","Data":"1f871f317f10497892e9d9717421f3d32dc211660ac10b5f4ddf007baffac969"}
Sep 15 21:41:25.289969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:25.289523    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9" event={"ID":"03ca6cea-ec0c-4d55-8b9f-02c74af881e4","Type":"ContainerStarted","Data":"97f3adec0bd1d5ccc3f445095f4c7cea7b884dff55e7ad77790b8508d492c819"}
Sep 15 21:41:25.309551 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:25.309500    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9" podStartSLOduration=2.309483304 podStartE2EDuration="2.309483304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:25.307554745 +0000 UTC m=+923.119563190" watchObservedRunningTime="2026-09-15 21:41:25.309483304 +0000 UTC m=+923.121491749"
Sep 15 21:41:26.127993 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:26.127960    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:26.132563 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:26.132541    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:26.294855 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:26.294828    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:26.295858 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:26.295840    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:27.446178 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:27.446145    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:27.446576 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:27.446403    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:27.458878 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:27.458857    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:28.304658 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.304629    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:41:28.307172 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.307143    2575 generic.go:356] "Generic (PLEG): container finished" podID="c4426644-b394-4272-b4d3-af22fd8f03d6" containerID="6b5ced23b38d6993dc60b9abf25e4942ea9b186949f595dbdcfb88a849e9f258" exitCode=0
Sep 15 21:41:28.307283 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.307186    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"
Sep 15 21:41:28.307283 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.307221    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc" event={"ID":"c4426644-b394-4272-b4d3-af22fd8f03d6","Type":"ContainerDied","Data":"6b5ced23b38d6993dc60b9abf25e4942ea9b186949f595dbdcfb88a849e9f258"}
Sep 15 21:41:28.307423 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.307285    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc" event={"ID":"c4426644-b394-4272-b4d3-af22fd8f03d6","Type":"ContainerDied","Data":"5676de0def1478faf5a49f8ef6310e8fb1cbe0db3b4b99d2c15e232af9e4d19b"}
Sep 15 21:41:28.307423 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.307318    2575 scope.go:117] "RemoveContainer" containerID="6b5ced23b38d6993dc60b9abf25e4942ea9b186949f595dbdcfb88a849e9f258"
Sep 15 21:41:28.316285 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.316259    2575 scope.go:117] "RemoveContainer" containerID="9f25257d3e45e5f0700ea02c32fcc60fc52b37197afd32958c797c2cdc958274"
Sep 15 21:41:28.319439 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.319421    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:41:28.327908 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.327462    2575 scope.go:117] "RemoveContainer" containerID="6b5ced23b38d6993dc60b9abf25e4942ea9b186949f595dbdcfb88a849e9f258"
Sep 15 21:41:28.328310 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:41:28.328082    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5ced23b38d6993dc60b9abf25e4942ea9b186949f595dbdcfb88a849e9f258\": container with ID starting with 6b5ced23b38d6993dc60b9abf25e4942ea9b186949f595dbdcfb88a849e9f258 not found: ID does not exist" containerID="6b5ced23b38d6993dc60b9abf25e4942ea9b186949f595dbdcfb88a849e9f258"
Sep 15 21:41:28.328310 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.328118    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5ced23b38d6993dc60b9abf25e4942ea9b186949f595dbdcfb88a849e9f258"} err="failed to get container status \"6b5ced23b38d6993dc60b9abf25e4942ea9b186949f595dbdcfb88a849e9f258\": rpc error: code = NotFound desc = could not find container \"6b5ced23b38d6993dc60b9abf25e4942ea9b186949f595dbdcfb88a849e9f258\": container with ID starting with 6b5ced23b38d6993dc60b9abf25e4942ea9b186949f595dbdcfb88a849e9f258 not found: ID does not exist"
Sep 15 21:41:28.328310 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.328139    2575 scope.go:117] "RemoveContainer" containerID="9f25257d3e45e5f0700ea02c32fcc60fc52b37197afd32958c797c2cdc958274"
Sep 15 21:41:28.328579 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:41:28.328438    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f25257d3e45e5f0700ea02c32fcc60fc52b37197afd32958c797c2cdc958274\": container with ID starting with 9f25257d3e45e5f0700ea02c32fcc60fc52b37197afd32958c797c2cdc958274 not found: ID does not exist" containerID="9f25257d3e45e5f0700ea02c32fcc60fc52b37197afd32958c797c2cdc958274"
Sep 15 21:41:28.328579 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.328463    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f25257d3e45e5f0700ea02c32fcc60fc52b37197afd32958c797c2cdc958274"} err="failed to get container status \"9f25257d3e45e5f0700ea02c32fcc60fc52b37197afd32958c797c2cdc958274\": rpc error: code = NotFound desc = could not find container \"9f25257d3e45e5f0700ea02c32fcc60fc52b37197afd32958c797c2cdc958274\": container with ID starting with 9f25257d3e45e5f0700ea02c32fcc60fc52b37197afd32958c797c2cdc958274 not found: ID does not exist"
Sep 15 21:41:28.389902 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.389870    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-token\") pod \"c4426644-b394-4272-b4d3-af22fd8f03d6\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") "
Sep 15 21:41:28.390097 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.389923    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-podinfo\") pod \"c4426644-b394-4272-b4d3-af22fd8f03d6\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") "
Sep 15 21:41:28.390097 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.389971    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4426644-b394-4272-b4d3-af22fd8f03d6-istiod-ca-cert\") pod \"c4426644-b394-4272-b4d3-af22fd8f03d6\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") "
Sep 15 21:41:28.390097 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.390008    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-envoy\") pod \"c4426644-b394-4272-b4d3-af22fd8f03d6\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") "
Sep 15 21:41:28.390097 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.390049    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-data\") pod \"c4426644-b394-4272-b4d3-af22fd8f03d6\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") "
Sep 15 21:41:28.390097 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.390077    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-workload-certs\") pod \"c4426644-b394-4272-b4d3-af22fd8f03d6\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") "
Sep 15 21:41:28.390374 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.390131    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tppkr\" (UniqueName: \"kubernetes.io/projected/c4426644-b394-4272-b4d3-af22fd8f03d6-kube-api-access-tppkr\") pod \"c4426644-b394-4272-b4d3-af22fd8f03d6\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") "
Sep 15 21:41:28.390374 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.390164    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-workload-socket\") pod \"c4426644-b394-4272-b4d3-af22fd8f03d6\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") "
Sep 15 21:41:28.390374 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.390221    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-credential-socket\") pod \"c4426644-b394-4272-b4d3-af22fd8f03d6\" (UID: \"c4426644-b394-4272-b4d3-af22fd8f03d6\") "
Sep 15 21:41:28.390374 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.390327    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-data" (OuterVolumeSpecName: "istio-data") pod "c4426644-b394-4272-b4d3-af22fd8f03d6" (UID: "c4426644-b394-4272-b4d3-af22fd8f03d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:41:28.390576 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.390402    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4426644-b394-4272-b4d3-af22fd8f03d6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c4426644-b394-4272-b4d3-af22fd8f03d6" (UID: "c4426644-b394-4272-b4d3-af22fd8f03d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:41:28.390576 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.390520    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c4426644-b394-4272-b4d3-af22fd8f03d6" (UID: "c4426644-b394-4272-b4d3-af22fd8f03d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:41:28.391305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.390786    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:41:28.391305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.390814    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:41:28.391305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.390829    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4426644-b394-4272-b4d3-af22fd8f03d6-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:41:28.391305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.391225    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c4426644-b394-4272-b4d3-af22fd8f03d6" (UID: "c4426644-b394-4272-b4d3-af22fd8f03d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:41:28.391305 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.391272    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c4426644-b394-4272-b4d3-af22fd8f03d6" (UID: "c4426644-b394-4272-b4d3-af22fd8f03d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:41:28.392483 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.392457    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c4426644-b394-4272-b4d3-af22fd8f03d6" (UID: "c4426644-b394-4272-b4d3-af22fd8f03d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:41:28.392787 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.392740    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c4426644-b394-4272-b4d3-af22fd8f03d6" (UID: "c4426644-b394-4272-b4d3-af22fd8f03d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:41:28.392881 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.392794    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-token" (OuterVolumeSpecName: "istio-token") pod "c4426644-b394-4272-b4d3-af22fd8f03d6" (UID: "c4426644-b394-4272-b4d3-af22fd8f03d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:41:28.393026 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.393009    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4426644-b394-4272-b4d3-af22fd8f03d6-kube-api-access-tppkr" (OuterVolumeSpecName: "kube-api-access-tppkr") pod "c4426644-b394-4272-b4d3-af22fd8f03d6" (UID: "c4426644-b394-4272-b4d3-af22fd8f03d6"). InnerVolumeSpecName "kube-api-access-tppkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:41:28.491723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.491691    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:41:28.491723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.491721    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tppkr\" (UniqueName: \"kubernetes.io/projected/c4426644-b394-4272-b4d3-af22fd8f03d6-kube-api-access-tppkr\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:41:28.492117 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.491732    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:41:28.492117 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.491741    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:41:28.492117 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.491749    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:41:28.492117 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.491772    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4426644-b394-4272-b4d3-af22fd8f03d6-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:41:28.642652 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.642621    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"]
Sep 15 21:41:28.648628 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.648607    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8316f91a-openshift-default-6c96656c98-9crcc"]
Sep 15 21:41:28.748710 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:28.748677    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4426644-b394-4272-b4d3-af22fd8f03d6" path="/var/lib/kubelet/pods/c4426644-b394-4272-b4d3-af22fd8f03d6/volumes"
Sep 15 21:41:31.866386 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:31.866355    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"]
Sep 15 21:41:31.866844 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:31.866830    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4426644-b394-4272-b4d3-af22fd8f03d6" containerName="istio-proxy"
Sep 15 21:41:31.866904 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:31.866849    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4426644-b394-4272-b4d3-af22fd8f03d6" containerName="istio-proxy"
Sep 15 21:41:31.866962 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:31.866952    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4426644-b394-4272-b4d3-af22fd8f03d6" containerName="istio-proxy"
Sep 15 21:41:31.870139 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:31.870123    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"
Sep 15 21:41:31.872264 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:31.872237    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-8ce29e8a\""
Sep 15 21:41:31.881022 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:31.880993    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"]
Sep 15 21:41:31.911259 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:31.911226    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"]
Sep 15 21:41:31.911688 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:31.911668    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4426644-b394-4272-b4d3-af22fd8f03d6" containerName="istio-proxy"
Sep 15 21:41:31.911688 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:31.911681    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4426644-b394-4272-b4d3-af22fd8f03d6" containerName="istio-proxy"
Sep 15 21:41:31.911800 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:31.911790    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4426644-b394-4272-b4d3-af22fd8f03d6" containerName="istio-proxy"
Sep 15 21:41:31.913959 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:31.913941    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"
Sep 15 21:41:31.916039 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:31.916018    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-8ce29e8a-dockercfg-hmcgq\""
Sep 15 21:41:31.923938 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:31.923912    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"]
Sep 15 21:41:32.027349 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:32.027316    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgvkt\" (UniqueName: \"kubernetes.io/projected/247b0b8f-aa1f-467a-8fb6-70ff9e7c0091-kube-api-access-wgvkt\") pod \"payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p\" (UID: \"247b0b8f-aa1f-467a-8fb6-70ff9e7c0091\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"
Sep 15 21:41:32.027349 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:32.027353    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26fbx\" (UniqueName: \"kubernetes.io/projected/44d66ff9-5e8e-429d-a4db-4edf90c01731-kube-api-access-26fbx\") pod \"payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn\" (UID: \"44d66ff9-5e8e-429d-a4db-4edf90c01731\") " pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"
Sep 15 21:41:32.027531 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:32.027377    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d66ff9-5e8e-429d-a4db-4edf90c01731-plugins-config-volume\") pod \"payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn\" (UID: \"44d66ff9-5e8e-429d-a4db-4edf90c01731\") " pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"
Sep 15 21:41:32.027531 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:32.027473    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247b0b8f-aa1f-467a-8fb6-70ff9e7c0091-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p\" (UID: \"247b0b8f-aa1f-467a-8fb6-70ff9e7c0091\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"
Sep 15 21:41:32.128996 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:32.128901    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgvkt\" (UniqueName: \"kubernetes.io/projected/247b0b8f-aa1f-467a-8fb6-70ff9e7c0091-kube-api-access-wgvkt\") pod \"payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p\" (UID: \"247b0b8f-aa1f-467a-8fb6-70ff9e7c0091\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"
Sep 15 21:41:32.128996 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:32.128945    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26fbx\" (UniqueName: \"kubernetes.io/projected/44d66ff9-5e8e-429d-a4db-4edf90c01731-kube-api-access-26fbx\") pod \"payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn\" (UID: \"44d66ff9-5e8e-429d-a4db-4edf90c01731\") " pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"
Sep 15 21:41:32.128996 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:32.128966    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d66ff9-5e8e-429d-a4db-4edf90c01731-plugins-config-volume\") pod \"payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn\" (UID: \"44d66ff9-5e8e-429d-a4db-4edf90c01731\") " pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"
Sep 15 21:41:32.128996 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:32.128995    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247b0b8f-aa1f-467a-8fb6-70ff9e7c0091-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p\" (UID: \"247b0b8f-aa1f-467a-8fb6-70ff9e7c0091\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"
Sep 15 21:41:32.129714 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:32.129685    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d66ff9-5e8e-429d-a4db-4edf90c01731-plugins-config-volume\") pod \"payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn\" (UID: \"44d66ff9-5e8e-429d-a4db-4edf90c01731\") " pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"
Sep 15 21:41:32.129793 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:32.129689    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247b0b8f-aa1f-467a-8fb6-70ff9e7c0091-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p\" (UID: \"247b0b8f-aa1f-467a-8fb6-70ff9e7c0091\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"
Sep 15 21:41:32.136570 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:32.136539    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgvkt\" (UniqueName: \"kubernetes.io/projected/247b0b8f-aa1f-467a-8fb6-70ff9e7c0091-kube-api-access-wgvkt\") pod \"payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p\" (UID: \"247b0b8f-aa1f-467a-8fb6-70ff9e7c0091\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"
Sep 15 21:41:32.136678 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:32.136589    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26fbx\" (UniqueName: \"kubernetes.io/projected/44d66ff9-5e8e-429d-a4db-4edf90c01731-kube-api-access-26fbx\") pod \"payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn\" (UID: \"44d66ff9-5e8e-429d-a4db-4edf90c01731\") " pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"
Sep 15 21:41:32.181864 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:32.181833    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"
Sep 15 21:41:32.225744 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:32.225704    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"
Sep 15 21:41:32.454638 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:32.454608    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"]
Sep 15 21:41:32.456025 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:41:32.455999    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247b0b8f_aa1f_467a_8fb6_70ff9e7c0091.slice/crio-6f4af965232485b18b7946d96b0cf239594c0e0d04ee511bca0bf9f519530bc1 WatchSource:0}: Error finding container 6f4af965232485b18b7946d96b0cf239594c0e0d04ee511bca0bf9f519530bc1: Status 404 returned error can't find the container with id 6f4af965232485b18b7946d96b0cf239594c0e0d04ee511bca0bf9f519530bc1
Sep 15 21:41:32.508218 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:32.508192    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"]
Sep 15 21:41:32.509656 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:41:32.509629    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d66ff9_5e8e_429d_a4db_4edf90c01731.slice/crio-d11b3d0c918aa3fbd3d84f081901fceecba06786ccc007e70959ad77c0036ab7 WatchSource:0}: Error finding container d11b3d0c918aa3fbd3d84f081901fceecba06786ccc007e70959ad77c0036ab7: Status 404 returned error can't find the container with id d11b3d0c918aa3fbd3d84f081901fceecba06786ccc007e70959ad77c0036ab7
Sep 15 21:41:33.335944 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:33.335913    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p" event={"ID":"247b0b8f-aa1f-467a-8fb6-70ff9e7c0091","Type":"ContainerStarted","Data":"d4db80d0ead0a542a178ce32ce3ae9dc6185e609ada927fe58d7a9c76919c9a6"}
Sep 15 21:41:33.336399 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:33.335959    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p" event={"ID":"247b0b8f-aa1f-467a-8fb6-70ff9e7c0091","Type":"ContainerStarted","Data":"6f4af965232485b18b7946d96b0cf239594c0e0d04ee511bca0bf9f519530bc1"}
Sep 15 21:41:33.336399 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:33.335997    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"
Sep 15 21:41:33.337644 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:33.337621    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn" event={"ID":"44d66ff9-5e8e-429d-a4db-4edf90c01731","Type":"ContainerStarted","Data":"911ccf968c18337f176f15f1fb1f1b01011fd2e3bbf10ad37db4407048babb4c"}
Sep 15 21:41:33.337796 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:33.337655    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn" event={"ID":"44d66ff9-5e8e-429d-a4db-4edf90c01731","Type":"ContainerStarted","Data":"d11b3d0c918aa3fbd3d84f081901fceecba06786ccc007e70959ad77c0036ab7"}
Sep 15 21:41:33.337796 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:33.337682    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"
Sep 15 21:41:33.352155 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:33.352112    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p" podStartSLOduration=2.352098641 podStartE2EDuration="2.352098641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:33.351238656 +0000 UTC m=+931.163247101" watchObservedRunningTime="2026-09-15 21:41:33.352098641 +0000 UTC m=+931.164107085"
Sep 15 21:41:44.344944 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:44.344916    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"
Sep 15 21:41:44.345367 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:44.344982    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"
Sep 15 21:41:44.360154 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:44.360089    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn" podStartSLOduration=13.360067748 podStartE2EDuration="13.360067748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:33.367279608 +0000 UTC m=+931.179288053" watchObservedRunningTime="2026-09-15 21:41:44.360067748 +0000 UTC m=+942.172076193"
Sep 15 21:41:47.424451 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:47.424417    2575 generic.go:356] "Generic (PLEG): container finished" podID="03ca6cea-ec0c-4d55-8b9f-02c74af881e4" containerID="1f871f317f10497892e9d9717421f3d32dc211660ac10b5f4ddf007baffac969" exitCode=0
Sep 15 21:41:47.424910 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:47.424490    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9" event={"ID":"03ca6cea-ec0c-4d55-8b9f-02c74af881e4","Type":"ContainerDied","Data":"1f871f317f10497892e9d9717421f3d32dc211660ac10b5f4ddf007baffac969"}
Sep 15 21:41:47.424974 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:47.424925    2575 scope.go:117] "RemoveContainer" containerID="1f871f317f10497892e9d9717421f3d32dc211660ac10b5f4ddf007baffac969"
Sep 15 21:41:47.426163 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:47.426130    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:41:47.426261 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:47.426195    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:41:47.426261 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:47.426225    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:41:48.432374 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:48.432338    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9" event={"ID":"03ca6cea-ec0c-4d55-8b9f-02c74af881e4","Type":"ContainerStarted","Data":"4b7049d7bad228f0452bea0cb876884a330350e1ba9d2303fa7e4a73f066b002"}
Sep 15 21:41:48.443825 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:48.443796    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:41:49.128247 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:49.128212    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:49.128437 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:49.128404    2575 patch_prober.go:28] interesting pod/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:41:49.128482 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:49.128453    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9" podUID="03ca6cea-ec0c-4d55-8b9f-02c74af881e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 15 21:41:50.127965 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:50.127924    2575 patch_prober.go:28] interesting pod/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:41:50.128381 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:50.127981    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9" podUID="03ca6cea-ec0c-4d55-8b9f-02c74af881e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 15 21:41:51.127820 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:51.127725    2575 patch_prober.go:28] interesting pod/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:41:51.128075 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:51.127845    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9" podUID="03ca6cea-ec0c-4d55-8b9f-02c74af881e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 15 21:41:52.127787 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:52.127708    2575 patch_prober.go:28] interesting pod/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:41:52.128004 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:52.127820    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9" podUID="03ca6cea-ec0c-4d55-8b9f-02c74af881e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 15 21:41:53.128108 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:53.128062    2575 patch_prober.go:28] interesting pod/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 15 21:41:53.128593 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:53.128132    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9" podUID="03ca6cea-ec0c-4d55-8b9f-02c74af881e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 15 21:41:54.185366 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:54.185288    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:54.185994 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:54.185784    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:54.186975 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:54.186946    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:41:55.130563 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.130516    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"]
Sep 15 21:41:55.134140 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.134110    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.136522 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.136496    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1af1db\"/\"openshift-service-ca.crt\""
Sep 15 21:41:55.137507 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.137482    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1af1db\"/\"default-dockercfg-4kjx2\""
Sep 15 21:41:55.137771 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.137596    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1af1db\"/\"kube-root-ca.crt\""
Sep 15 21:41:55.137771 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.137488    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1af1db\"/\"test-model-1af1db-kserve-self-signed-certs\""
Sep 15 21:41:55.142218 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.142184    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"]
Sep 15 21:41:55.184679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.184640    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-model-cache\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.184679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.184678    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-tmp-dir\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.184963 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.184708    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-dshm\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.184963 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.184804    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-home\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.184963 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.184841    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8fsz\" (UniqueName: \"kubernetes.io/projected/10dac161-67f3-44ab-b402-092ed04cf803-kube-api-access-l8fsz\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.185087 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.184976    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.286173 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.286124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.286173 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.286176    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-model-cache\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.286736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.286192    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-tmp-dir\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.286736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.286220    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-dshm\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.286736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.286247    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-home\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.286736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.286274    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8fsz\" (UniqueName: \"kubernetes.io/projected/10dac161-67f3-44ab-b402-092ed04cf803-kube-api-access-l8fsz\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.286736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.286717    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-model-cache\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.286981 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.286744    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-tmp-dir\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.286981 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.286815    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-home\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.288850 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.288826    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-dshm\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.289130 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.289110    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.295630 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.295588    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8fsz\" (UniqueName: \"kubernetes.io/projected/10dac161-67f3-44ab-b402-092ed04cf803-kube-api-access-l8fsz\") pod \"test-model-1af1db-kserve-68f79b48f9-fj2qk\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") " pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.447129 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.447024    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:41:55.734660 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:55.734609    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"]
Sep 15 21:41:55.738112 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:41:55.738064    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10dac161_67f3_44ab_b402_092ed04cf803.slice/crio-2f06508d6362e7fa2ed997ad7c25067b4ac3e904b6a252d782f03ef6803f9d93 WatchSource:0}: Error finding container 2f06508d6362e7fa2ed997ad7c25067b4ac3e904b6a252d782f03ef6803f9d93: Status 404 returned error can't find the container with id 2f06508d6362e7fa2ed997ad7c25067b4ac3e904b6a252d782f03ef6803f9d93
Sep 15 21:41:56.494129 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:56.494089    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk" event={"ID":"10dac161-67f3-44ab-b402-092ed04cf803","Type":"ContainerStarted","Data":"7e4cb359b66f42c195a66500b21e6af41d18c60bba03f1019f31f0e488fcb630"}
Sep 15 21:41:56.494554 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:56.494145    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk" event={"ID":"10dac161-67f3-44ab-b402-092ed04cf803","Type":"ContainerStarted","Data":"2f06508d6362e7fa2ed997ad7c25067b4ac3e904b6a252d782f03ef6803f9d93"}
Sep 15 21:41:56.511476 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:56.511418    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk" podStartSLOduration=1.51139988 podStartE2EDuration="1.51139988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:56.510155242 +0000 UTC m=+954.322163686" watchObservedRunningTime="2026-09-15 21:41:56.51139988 +0000 UTC m=+954.323408323"
Sep 15 21:41:57.315262 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:57.315218    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4"]
Sep 15 21:41:57.318154 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:57.318124    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4"
Sep 15 21:41:57.320400 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:57.320373    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:41:57.320589 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:57.320417    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 21:41:57.326430 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:57.326389    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4"]
Sep 15 21:41:57.406789 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:57.406713    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6720b1d-786e-4d70-b532-cf6690e1d873-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4\" (UID: \"b6720b1d-786e-4d70-b532-cf6690e1d873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4"
Sep 15 21:41:57.508323 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:57.508275    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6720b1d-786e-4d70-b532-cf6690e1d873-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4\" (UID: \"b6720b1d-786e-4d70-b532-cf6690e1d873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4"
Sep 15 21:41:57.509055 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:57.509031    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6720b1d-786e-4d70-b532-cf6690e1d873-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4\" (UID: \"b6720b1d-786e-4d70-b532-cf6690e1d873\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4"
Sep 15 21:41:57.632308 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:57.632274    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4"
Sep 15 21:41:57.908817 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:57.908789    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4"]
Sep 15 21:41:57.911042 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:41:57.911005    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6720b1d_786e_4d70_b532_cf6690e1d873.slice/crio-0fd21e51ae82dac872af31e5b1be3c9e14cf6fb1f66a9c9c2cf9767cdc86dd6a WatchSource:0}: Error finding container 0fd21e51ae82dac872af31e5b1be3c9e14cf6fb1f66a9c9c2cf9767cdc86dd6a: Status 404 returned error can't find the container with id 0fd21e51ae82dac872af31e5b1be3c9e14cf6fb1f66a9c9c2cf9767cdc86dd6a
Sep 15 21:41:58.509529 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:58.509426    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4" event={"ID":"b6720b1d-786e-4d70-b532-cf6690e1d873","Type":"ContainerStarted","Data":"8e842d95c7c2808f5f4f5fec9d73ecd1d79ff3e26c5f2d8babe2a25c76137158"}
Sep 15 21:41:58.509969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:58.509538    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4" event={"ID":"b6720b1d-786e-4d70-b532-cf6690e1d873","Type":"ContainerStarted","Data":"0fd21e51ae82dac872af31e5b1be3c9e14cf6fb1f66a9c9c2cf9767cdc86dd6a"}
Sep 15 21:41:58.538888 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:58.538821    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4" podStartSLOduration=1.538799473 podStartE2EDuration="1.538799473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:41:58.536517085 +0000 UTC m=+956.348525528" watchObservedRunningTime="2026-09-15 21:41:58.538799473 +0000 UTC m=+956.350807917"
Sep 15 21:41:59.517868 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:59.517822    2575 generic.go:356] "Generic (PLEG): container finished" podID="b6720b1d-786e-4d70-b532-cf6690e1d873" containerID="8e842d95c7c2808f5f4f5fec9d73ecd1d79ff3e26c5f2d8babe2a25c76137158" exitCode=0
Sep 15 21:41:59.518304 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:41:59.517910    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4" event={"ID":"b6720b1d-786e-4d70-b532-cf6690e1d873","Type":"ContainerDied","Data":"8e842d95c7c2808f5f4f5fec9d73ecd1d79ff3e26c5f2d8babe2a25c76137158"}
Sep 15 21:42:00.671190 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:00.671155    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4"
Sep 15 21:42:00.748166 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:00.748132    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6720b1d-786e-4d70-b532-cf6690e1d873-maas-api-service-ca\") pod \"b6720b1d-786e-4d70-b532-cf6690e1d873\" (UID: \"b6720b1d-786e-4d70-b532-cf6690e1d873\") "
Sep 15 21:42:00.748541 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:00.748516    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6720b1d-786e-4d70-b532-cf6690e1d873-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b6720b1d-786e-4d70-b532-cf6690e1d873" (UID: "b6720b1d-786e-4d70-b532-cf6690e1d873"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:42:00.850215 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:00.850179    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6720b1d-786e-4d70-b532-cf6690e1d873-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:01.531564 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:01.531532    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4"
Sep 15 21:42:01.531803 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:01.531577    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4" event={"ID":"b6720b1d-786e-4d70-b532-cf6690e1d873","Type":"ContainerDied","Data":"0fd21e51ae82dac872af31e5b1be3c9e14cf6fb1f66a9c9c2cf9767cdc86dd6a"}
Sep 15 21:42:01.531803 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:01.531625    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fd21e51ae82dac872af31e5b1be3c9e14cf6fb1f66a9c9c2cf9767cdc86dd6a"
Sep 15 21:42:04.336108 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:04.336059    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:42:05.447966 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:05.447922    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:42:05.448400 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:05.447981    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:42:05.461323 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:05.461291    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:42:05.568262 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:05.568227    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:42:09.171121 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:09.171068    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5416b7-w9dsz"]
Sep 15 21:42:09.171618 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:09.171602    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6720b1d-786e-4d70-b532-cf6690e1d873" containerName="revoke-keys"
Sep 15 21:42:09.171663 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:09.171620    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6720b1d-786e-4d70-b532-cf6690e1d873" containerName="revoke-keys"
Sep 15 21:42:09.171700 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:09.171692    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6720b1d-786e-4d70-b532-cf6690e1d873" containerName="revoke-keys"
Sep 15 21:42:09.176280 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:09.176251    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5416b7-w9dsz"
Sep 15 21:42:09.178670 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:09.178571    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:42:09.179611 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:09.179583    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 21:42:09.183315 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:09.183274    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5416b7-w9dsz"]
Sep 15 21:42:09.349720 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:09.349670    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03d0f17-0241-41b4-af38-568e50659d5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5416b7-w9dsz\" (UID: \"e03d0f17-0241-41b4-af38-568e50659d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5416b7-w9dsz"
Sep 15 21:42:09.451419 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:09.451294    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03d0f17-0241-41b4-af38-568e50659d5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5416b7-w9dsz\" (UID: \"e03d0f17-0241-41b4-af38-568e50659d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5416b7-w9dsz"
Sep 15 21:42:09.452052 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:09.452027    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03d0f17-0241-41b4-af38-568e50659d5a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-5416b7-w9dsz\" (UID: \"e03d0f17-0241-41b4-af38-568e50659d5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5416b7-w9dsz"
Sep 15 21:42:09.492351 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:09.492304    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5416b7-w9dsz"
Sep 15 21:42:09.768953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:09.768922    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5416b7-w9dsz"]
Sep 15 21:42:09.771077 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:42:09.771042    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode03d0f17_0241_41b4_af38_568e50659d5a.slice/crio-a8a3dc66295962b7a1f9b9b6355bbb26b2be12b741a658103601f7bb3023e168 WatchSource:0}: Error finding container a8a3dc66295962b7a1f9b9b6355bbb26b2be12b741a658103601f7bb3023e168: Status 404 returned error can't find the container with id a8a3dc66295962b7a1f9b9b6355bbb26b2be12b741a658103601f7bb3023e168
Sep 15 21:42:10.588941 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:10.588904    2575 generic.go:356] "Generic (PLEG): container finished" podID="e03d0f17-0241-41b4-af38-568e50659d5a" containerID="c44cb25299cab0c021782025f374d04c6d3d09dcccc0fae2b30fa76b030fd0e6" exitCode=0
Sep 15 21:42:10.589312 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:10.588989    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5416b7-w9dsz" event={"ID":"e03d0f17-0241-41b4-af38-568e50659d5a","Type":"ContainerDied","Data":"c44cb25299cab0c021782025f374d04c6d3d09dcccc0fae2b30fa76b030fd0e6"}
Sep 15 21:42:10.589312 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:10.589047    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5416b7-w9dsz" event={"ID":"e03d0f17-0241-41b4-af38-568e50659d5a","Type":"ContainerStarted","Data":"a8a3dc66295962b7a1f9b9b6355bbb26b2be12b741a658103601f7bb3023e168"}
Sep 15 21:42:11.749537 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:11.749498    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5416b7-w9dsz"
Sep 15 21:42:11.775851 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:11.775802    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03d0f17-0241-41b4-af38-568e50659d5a-maas-api-service-ca\") pod \"e03d0f17-0241-41b4-af38-568e50659d5a\" (UID: \"e03d0f17-0241-41b4-af38-568e50659d5a\") "
Sep 15 21:42:11.776209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:11.776183    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e03d0f17-0241-41b4-af38-568e50659d5a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e03d0f17-0241-41b4-af38-568e50659d5a" (UID: "e03d0f17-0241-41b4-af38-568e50659d5a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:42:11.776324 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:11.776282    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e03d0f17-0241-41b4-af38-568e50659d5a-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:12.603209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:12.603172    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5416b7-w9dsz" event={"ID":"e03d0f17-0241-41b4-af38-568e50659d5a","Type":"ContainerDied","Data":"a8a3dc66295962b7a1f9b9b6355bbb26b2be12b741a658103601f7bb3023e168"}
Sep 15 21:42:12.603429 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:12.603235    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8a3dc66295962b7a1f9b9b6355bbb26b2be12b741a658103601f7bb3023e168"
Sep 15 21:42:12.603429 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:12.603259    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5416b7-w9dsz"
Sep 15 21:42:13.941558 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:13.941472    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"]
Sep 15 21:42:13.942111 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:13.941838    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn" podUID="44d66ff9-5e8e-429d-a4db-4edf90c01731" containerName="payload-processing" containerID="cri-o://911ccf968c18337f176f15f1fb1f1b01011fd2e3bbf10ad37db4407048babb4c" gracePeriod=30
Sep 15 21:42:13.946857 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:13.946816    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"]
Sep 15 21:42:13.947370 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:13.947329    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p" podUID="247b0b8f-aa1f-467a-8fb6-70ff9e7c0091" containerName="payload-pre-processing" containerID="cri-o://d4db80d0ead0a542a178ce32ce3ae9dc6185e609ada927fe58d7a9c76919c9a6" gracePeriod=30
Sep 15 21:42:14.101164 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.100628    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4"]
Sep 15 21:42:14.104711 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.104670    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-8ce29e8a-qfzj4"]
Sep 15 21:42:14.241939 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.241892    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"]
Sep 15 21:42:14.242176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.242149    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9" podUID="03ca6cea-ec0c-4d55-8b9f-02c74af881e4" containerName="istio-proxy" containerID="cri-o://4b7049d7bad228f0452bea0cb876884a330350e1ba9d2303fa7e4a73f066b002" gracePeriod=30
Sep 15 21:42:14.254953 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.254923    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"
Sep 15 21:42:14.258454 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.258426    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"
Sep 15 21:42:14.300345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.300304    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgvkt\" (UniqueName: \"kubernetes.io/projected/247b0b8f-aa1f-467a-8fb6-70ff9e7c0091-kube-api-access-wgvkt\") pod \"247b0b8f-aa1f-467a-8fb6-70ff9e7c0091\" (UID: \"247b0b8f-aa1f-467a-8fb6-70ff9e7c0091\") "
Sep 15 21:42:14.300607 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.300380    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247b0b8f-aa1f-467a-8fb6-70ff9e7c0091-plugins-config-volume\") pod \"247b0b8f-aa1f-467a-8fb6-70ff9e7c0091\" (UID: \"247b0b8f-aa1f-467a-8fb6-70ff9e7c0091\") "
Sep 15 21:42:14.300607 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.300438    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26fbx\" (UniqueName: \"kubernetes.io/projected/44d66ff9-5e8e-429d-a4db-4edf90c01731-kube-api-access-26fbx\") pod \"44d66ff9-5e8e-429d-a4db-4edf90c01731\" (UID: \"44d66ff9-5e8e-429d-a4db-4edf90c01731\") "
Sep 15 21:42:14.300607 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.300463    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d66ff9-5e8e-429d-a4db-4edf90c01731-plugins-config-volume\") pod \"44d66ff9-5e8e-429d-a4db-4edf90c01731\" (UID: \"44d66ff9-5e8e-429d-a4db-4edf90c01731\") "
Sep 15 21:42:14.301105 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.300868    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/247b0b8f-aa1f-467a-8fb6-70ff9e7c0091-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "247b0b8f-aa1f-467a-8fb6-70ff9e7c0091" (UID: "247b0b8f-aa1f-467a-8fb6-70ff9e7c0091"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:42:14.301168 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.301028    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/247b0b8f-aa1f-467a-8fb6-70ff9e7c0091-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:14.301270 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.301227    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d66ff9-5e8e-429d-a4db-4edf90c01731-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44d66ff9-5e8e-429d-a4db-4edf90c01731" (UID: "44d66ff9-5e8e-429d-a4db-4edf90c01731"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:42:14.303465 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.303421    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d66ff9-5e8e-429d-a4db-4edf90c01731-kube-api-access-26fbx" (OuterVolumeSpecName: "kube-api-access-26fbx") pod "44d66ff9-5e8e-429d-a4db-4edf90c01731" (UID: "44d66ff9-5e8e-429d-a4db-4edf90c01731"). InnerVolumeSpecName "kube-api-access-26fbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:42:14.303614 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.303552    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/247b0b8f-aa1f-467a-8fb6-70ff9e7c0091-kube-api-access-wgvkt" (OuterVolumeSpecName: "kube-api-access-wgvkt") pod "247b0b8f-aa1f-467a-8fb6-70ff9e7c0091" (UID: "247b0b8f-aa1f-467a-8fb6-70ff9e7c0091"). InnerVolumeSpecName "kube-api-access-wgvkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:42:14.402079 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.402030    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgvkt\" (UniqueName: \"kubernetes.io/projected/247b0b8f-aa1f-467a-8fb6-70ff9e7c0091-kube-api-access-wgvkt\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:14.402079 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.402071    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26fbx\" (UniqueName: \"kubernetes.io/projected/44d66ff9-5e8e-429d-a4db-4edf90c01731-kube-api-access-26fbx\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:14.402079 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.402085    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44d66ff9-5e8e-429d-a4db-4edf90c01731-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:14.617794 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.617736    2575 generic.go:356] "Generic (PLEG): container finished" podID="247b0b8f-aa1f-467a-8fb6-70ff9e7c0091" containerID="d4db80d0ead0a542a178ce32ce3ae9dc6185e609ada927fe58d7a9c76919c9a6" exitCode=0
Sep 15 21:42:14.617794 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.617786    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p" event={"ID":"247b0b8f-aa1f-467a-8fb6-70ff9e7c0091","Type":"ContainerDied","Data":"d4db80d0ead0a542a178ce32ce3ae9dc6185e609ada927fe58d7a9c76919c9a6"}
Sep 15 21:42:14.618068 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.617802    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"
Sep 15 21:42:14.618068 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.617845    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p" event={"ID":"247b0b8f-aa1f-467a-8fb6-70ff9e7c0091","Type":"ContainerDied","Data":"6f4af965232485b18b7946d96b0cf239594c0e0d04ee511bca0bf9f519530bc1"}
Sep 15 21:42:14.618068 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.617868    2575 scope.go:117] "RemoveContainer" containerID="d4db80d0ead0a542a178ce32ce3ae9dc6185e609ada927fe58d7a9c76919c9a6"
Sep 15 21:42:14.619960 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.619898    2575 generic.go:356] "Generic (PLEG): container finished" podID="44d66ff9-5e8e-429d-a4db-4edf90c01731" containerID="911ccf968c18337f176f15f1fb1f1b01011fd2e3bbf10ad37db4407048babb4c" exitCode=0
Sep 15 21:42:14.619960 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.619935    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn" event={"ID":"44d66ff9-5e8e-429d-a4db-4edf90c01731","Type":"ContainerDied","Data":"911ccf968c18337f176f15f1fb1f1b01011fd2e3bbf10ad37db4407048babb4c"}
Sep 15 21:42:14.619960 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.619956    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"
Sep 15 21:42:14.620157 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.619993    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn" event={"ID":"44d66ff9-5e8e-429d-a4db-4edf90c01731","Type":"ContainerDied","Data":"d11b3d0c918aa3fbd3d84f081901fceecba06786ccc007e70959ad77c0036ab7"}
Sep 15 21:42:14.629133 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.629103    2575 scope.go:117] "RemoveContainer" containerID="d4db80d0ead0a542a178ce32ce3ae9dc6185e609ada927fe58d7a9c76919c9a6"
Sep 15 21:42:14.629579 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:42:14.629546    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4db80d0ead0a542a178ce32ce3ae9dc6185e609ada927fe58d7a9c76919c9a6\": container with ID starting with d4db80d0ead0a542a178ce32ce3ae9dc6185e609ada927fe58d7a9c76919c9a6 not found: ID does not exist" containerID="d4db80d0ead0a542a178ce32ce3ae9dc6185e609ada927fe58d7a9c76919c9a6"
Sep 15 21:42:14.629655 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.629593    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4db80d0ead0a542a178ce32ce3ae9dc6185e609ada927fe58d7a9c76919c9a6"} err="failed to get container status \"d4db80d0ead0a542a178ce32ce3ae9dc6185e609ada927fe58d7a9c76919c9a6\": rpc error: code = NotFound desc = could not find container \"d4db80d0ead0a542a178ce32ce3ae9dc6185e609ada927fe58d7a9c76919c9a6\": container with ID starting with d4db80d0ead0a542a178ce32ce3ae9dc6185e609ada927fe58d7a9c76919c9a6 not found: ID does not exist"
Sep 15 21:42:14.629655 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.629619    2575 scope.go:117] "RemoveContainer" containerID="911ccf968c18337f176f15f1fb1f1b01011fd2e3bbf10ad37db4407048babb4c"
Sep 15 21:42:14.640185 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.640156    2575 scope.go:117] "RemoveContainer" containerID="911ccf968c18337f176f15f1fb1f1b01011fd2e3bbf10ad37db4407048babb4c"
Sep 15 21:42:14.640564 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:42:14.640542    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"911ccf968c18337f176f15f1fb1f1b01011fd2e3bbf10ad37db4407048babb4c\": container with ID starting with 911ccf968c18337f176f15f1fb1f1b01011fd2e3bbf10ad37db4407048babb4c not found: ID does not exist" containerID="911ccf968c18337f176f15f1fb1f1b01011fd2e3bbf10ad37db4407048babb4c"
Sep 15 21:42:14.640632 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.640575    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"911ccf968c18337f176f15f1fb1f1b01011fd2e3bbf10ad37db4407048babb4c"} err="failed to get container status \"911ccf968c18337f176f15f1fb1f1b01011fd2e3bbf10ad37db4407048babb4c\": rpc error: code = NotFound desc = could not find container \"911ccf968c18337f176f15f1fb1f1b01011fd2e3bbf10ad37db4407048babb4c\": container with ID starting with 911ccf968c18337f176f15f1fb1f1b01011fd2e3bbf10ad37db4407048babb4c not found: ID does not exist"
Sep 15 21:42:14.646574 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.646528    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"]
Sep 15 21:42:14.650184 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.650152    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8ce29e8a-7d48d64b8b-5xn5p"]
Sep 15 21:42:14.658711 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.658669    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"]
Sep 15 21:42:14.664176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.664142    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8ce29e8a-6db86685-qvdvn"]
Sep 15 21:42:14.749815 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.749774    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="247b0b8f-aa1f-467a-8fb6-70ff9e7c0091" path="/var/lib/kubelet/pods/247b0b8f-aa1f-467a-8fb6-70ff9e7c0091/volumes"
Sep 15 21:42:14.750202 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.750187    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44d66ff9-5e8e-429d-a4db-4edf90c01731" path="/var/lib/kubelet/pods/44d66ff9-5e8e-429d-a4db-4edf90c01731/volumes"
Sep 15 21:42:14.750513 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:14.750500    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6720b1d-786e-4d70-b532-cf6690e1d873" path="/var/lib/kubelet/pods/b6720b1d-786e-4d70-b532-cf6690e1d873/volumes"
Sep 15 21:42:19.527419 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.527385    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:42:19.554209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.554171    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-credential-socket\") pod \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") "
Sep 15 21:42:19.554391 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.554243    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-workload-socket\") pod \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") "
Sep 15 21:42:19.554391 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.554276    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-envoy\") pod \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") "
Sep 15 21:42:19.554391 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.554329    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-token\") pod \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") "
Sep 15 21:42:19.554391 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.554385    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istiod-ca-cert\") pod \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") "
Sep 15 21:42:19.554586 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.554551    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "03ca6cea-ec0c-4d55-8b9f-02c74af881e4" (UID: "03ca6cea-ec0c-4d55-8b9f-02c74af881e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:42:19.554635 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.554556    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-data\") pod \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") "
Sep 15 21:42:19.554690 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.554631    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "03ca6cea-ec0c-4d55-8b9f-02c74af881e4" (UID: "03ca6cea-ec0c-4d55-8b9f-02c74af881e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:42:19.554690 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.554684    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-podinfo\") pod \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") "
Sep 15 21:42:19.554825 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.554743    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-workload-certs\") pod \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") "
Sep 15 21:42:19.554825 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.554787    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhdj2\" (UniqueName: \"kubernetes.io/projected/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-kube-api-access-vhdj2\") pod \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\" (UID: \"03ca6cea-ec0c-4d55-8b9f-02c74af881e4\") "
Sep 15 21:42:19.554935 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.554844    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-data" (OuterVolumeSpecName: "istio-data") pod "03ca6cea-ec0c-4d55-8b9f-02c74af881e4" (UID: "03ca6cea-ec0c-4d55-8b9f-02c74af881e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:42:19.555261 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.555040    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "03ca6cea-ec0c-4d55-8b9f-02c74af881e4" (UID: "03ca6cea-ec0c-4d55-8b9f-02c74af881e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:42:19.555261 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.555215    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "03ca6cea-ec0c-4d55-8b9f-02c74af881e4" (UID: "03ca6cea-ec0c-4d55-8b9f-02c74af881e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:42:19.555621 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.555401    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:19.555621 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.555422    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:19.555621 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.555437    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:19.555621 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.555453    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:19.557099 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.557066    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-token" (OuterVolumeSpecName: "istio-token") pod "03ca6cea-ec0c-4d55-8b9f-02c74af881e4" (UID: "03ca6cea-ec0c-4d55-8b9f-02c74af881e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:42:19.557221 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.557115    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "03ca6cea-ec0c-4d55-8b9f-02c74af881e4" (UID: "03ca6cea-ec0c-4d55-8b9f-02c74af881e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:42:19.557286 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.557267    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "03ca6cea-ec0c-4d55-8b9f-02c74af881e4" (UID: "03ca6cea-ec0c-4d55-8b9f-02c74af881e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:42:19.557324 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.557296    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-kube-api-access-vhdj2" (OuterVolumeSpecName: "kube-api-access-vhdj2") pod "03ca6cea-ec0c-4d55-8b9f-02c74af881e4" (UID: "03ca6cea-ec0c-4d55-8b9f-02c74af881e4"). InnerVolumeSpecName "kube-api-access-vhdj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:42:19.656122 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.656089    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:19.656122 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.656122    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:19.656420 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.656135    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:19.656420 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.656146    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:19.656420 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.656155    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhdj2\" (UniqueName: \"kubernetes.io/projected/03ca6cea-ec0c-4d55-8b9f-02c74af881e4-kube-api-access-vhdj2\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:19.657002 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.656968    2575 generic.go:356] "Generic (PLEG): container finished" podID="03ca6cea-ec0c-4d55-8b9f-02c74af881e4" containerID="4b7049d7bad228f0452bea0cb876884a330350e1ba9d2303fa7e4a73f066b002" exitCode=0
Sep 15 21:42:19.657092 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.657077    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"
Sep 15 21:42:19.657182 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.657105    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9" event={"ID":"03ca6cea-ec0c-4d55-8b9f-02c74af881e4","Type":"ContainerDied","Data":"4b7049d7bad228f0452bea0cb876884a330350e1ba9d2303fa7e4a73f066b002"}
Sep 15 21:42:19.657182 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.657166    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9" event={"ID":"03ca6cea-ec0c-4d55-8b9f-02c74af881e4","Type":"ContainerDied","Data":"97f3adec0bd1d5ccc3f445095f4c7cea7b884dff55e7ad77790b8508d492c819"}
Sep 15 21:42:19.657248 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.657189    2575 scope.go:117] "RemoveContainer" containerID="4b7049d7bad228f0452bea0cb876884a330350e1ba9d2303fa7e4a73f066b002"
Sep 15 21:42:19.668711 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.668691    2575 scope.go:117] "RemoveContainer" containerID="1f871f317f10497892e9d9717421f3d32dc211660ac10b5f4ddf007baffac969"
Sep 15 21:42:19.680012 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.679978    2575 scope.go:117] "RemoveContainer" containerID="4b7049d7bad228f0452bea0cb876884a330350e1ba9d2303fa7e4a73f066b002"
Sep 15 21:42:19.680563 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:42:19.680530    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b7049d7bad228f0452bea0cb876884a330350e1ba9d2303fa7e4a73f066b002\": container with ID starting with 4b7049d7bad228f0452bea0cb876884a330350e1ba9d2303fa7e4a73f066b002 not found: ID does not exist" containerID="4b7049d7bad228f0452bea0cb876884a330350e1ba9d2303fa7e4a73f066b002"
Sep 15 21:42:19.680678 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.680580    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b7049d7bad228f0452bea0cb876884a330350e1ba9d2303fa7e4a73f066b002"} err="failed to get container status \"4b7049d7bad228f0452bea0cb876884a330350e1ba9d2303fa7e4a73f066b002\": rpc error: code = NotFound desc = could not find container \"4b7049d7bad228f0452bea0cb876884a330350e1ba9d2303fa7e4a73f066b002\": container with ID starting with 4b7049d7bad228f0452bea0cb876884a330350e1ba9d2303fa7e4a73f066b002 not found: ID does not exist"
Sep 15 21:42:19.680678 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.680609    2575 scope.go:117] "RemoveContainer" containerID="1f871f317f10497892e9d9717421f3d32dc211660ac10b5f4ddf007baffac969"
Sep 15 21:42:19.681606 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:42:19.681185    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f871f317f10497892e9d9717421f3d32dc211660ac10b5f4ddf007baffac969\": container with ID starting with 1f871f317f10497892e9d9717421f3d32dc211660ac10b5f4ddf007baffac969 not found: ID does not exist" containerID="1f871f317f10497892e9d9717421f3d32dc211660ac10b5f4ddf007baffac969"
Sep 15 21:42:19.681606 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.681223    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f871f317f10497892e9d9717421f3d32dc211660ac10b5f4ddf007baffac969"} err="failed to get container status \"1f871f317f10497892e9d9717421f3d32dc211660ac10b5f4ddf007baffac969\": rpc error: code = NotFound desc = could not find container \"1f871f317f10497892e9d9717421f3d32dc211660ac10b5f4ddf007baffac969\": container with ID starting with 1f871f317f10497892e9d9717421f3d32dc211660ac10b5f4ddf007baffac969 not found: ID does not exist"
Sep 15 21:42:19.682256 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.682199    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"]
Sep 15 21:42:19.686121 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.686087    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-8ce29e8a-openshift-default-577bbbb59b-mtft9"]
Sep 15 21:42:19.693904 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.693867    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"]
Sep 15 21:42:19.694150 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.694124    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr" podUID="7880acb1-c1d9-4509-ae56-1cd81a2ca6c8" containerName="payload-processing" containerID="cri-o://773cac1d44fa13f24e02ab7df85e5b924902de47c6c15f565ddab7f58a7ac557" gracePeriod=30
Sep 15 21:42:19.704735 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.704619    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"]
Sep 15 21:42:19.705452 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.705040    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw" podUID="71381e8f-030b-4482-a8fc-1a1133365237" containerName="payload-pre-processing" containerID="cri-o://c9a830540fdeb5804c2c80915e2669fa982fb599d64ca3fe3610543146fe371a" gracePeriod=30
Sep 15 21:42:19.849930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.849892    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5416b7-w9dsz"]
Sep 15 21:42:19.852100 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.852068    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5416b7-w9dsz"]
Sep 15 21:42:19.957555 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:19.957516    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"
Sep 15 21:42:20.001561 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.001528    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"
Sep 15 21:42:20.060399 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.060354    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71381e8f-030b-4482-a8fc-1a1133365237-plugins-config-volume\") pod \"71381e8f-030b-4482-a8fc-1a1133365237\" (UID: \"71381e8f-030b-4482-a8fc-1a1133365237\") "
Sep 15 21:42:20.060645 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.060505    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4rdk\" (UniqueName: \"kubernetes.io/projected/71381e8f-030b-4482-a8fc-1a1133365237-kube-api-access-h4rdk\") pod \"71381e8f-030b-4482-a8fc-1a1133365237\" (UID: \"71381e8f-030b-4482-a8fc-1a1133365237\") "
Sep 15 21:42:20.060645 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.060562    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mvc9\" (UniqueName: \"kubernetes.io/projected/7880acb1-c1d9-4509-ae56-1cd81a2ca6c8-kube-api-access-7mvc9\") pod \"7880acb1-c1d9-4509-ae56-1cd81a2ca6c8\" (UID: \"7880acb1-c1d9-4509-ae56-1cd81a2ca6c8\") "
Sep 15 21:42:20.060784 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.060646    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7880acb1-c1d9-4509-ae56-1cd81a2ca6c8-plugins-config-volume\") pod \"7880acb1-c1d9-4509-ae56-1cd81a2ca6c8\" (UID: \"7880acb1-c1d9-4509-ae56-1cd81a2ca6c8\") "
Sep 15 21:42:20.060784 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.060744    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71381e8f-030b-4482-a8fc-1a1133365237-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71381e8f-030b-4482-a8fc-1a1133365237" (UID: "71381e8f-030b-4482-a8fc-1a1133365237"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:42:20.061059 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.061033    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71381e8f-030b-4482-a8fc-1a1133365237-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:20.061179 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.061088    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7880acb1-c1d9-4509-ae56-1cd81a2ca6c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7880acb1-c1d9-4509-ae56-1cd81a2ca6c8" (UID: "7880acb1-c1d9-4509-ae56-1cd81a2ca6c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:42:20.063088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.063026    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7880acb1-c1d9-4509-ae56-1cd81a2ca6c8-kube-api-access-7mvc9" (OuterVolumeSpecName: "kube-api-access-7mvc9") pod "7880acb1-c1d9-4509-ae56-1cd81a2ca6c8" (UID: "7880acb1-c1d9-4509-ae56-1cd81a2ca6c8"). InnerVolumeSpecName "kube-api-access-7mvc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:42:20.063088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.063035    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71381e8f-030b-4482-a8fc-1a1133365237-kube-api-access-h4rdk" (OuterVolumeSpecName: "kube-api-access-h4rdk") pod "71381e8f-030b-4482-a8fc-1a1133365237" (UID: "71381e8f-030b-4482-a8fc-1a1133365237"). InnerVolumeSpecName "kube-api-access-h4rdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:42:20.162562 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.162515    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4rdk\" (UniqueName: \"kubernetes.io/projected/71381e8f-030b-4482-a8fc-1a1133365237-kube-api-access-h4rdk\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:20.162562 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.162556    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mvc9\" (UniqueName: \"kubernetes.io/projected/7880acb1-c1d9-4509-ae56-1cd81a2ca6c8-kube-api-access-7mvc9\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:20.162562 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.162566    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7880acb1-c1d9-4509-ae56-1cd81a2ca6c8-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:42:20.663981 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.663943    2575 generic.go:356] "Generic (PLEG): container finished" podID="71381e8f-030b-4482-a8fc-1a1133365237" containerID="c9a830540fdeb5804c2c80915e2669fa982fb599d64ca3fe3610543146fe371a" exitCode=0
Sep 15 21:42:20.664484 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.664004    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"
Sep 15 21:42:20.664484 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.664044    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw" event={"ID":"71381e8f-030b-4482-a8fc-1a1133365237","Type":"ContainerDied","Data":"c9a830540fdeb5804c2c80915e2669fa982fb599d64ca3fe3610543146fe371a"}
Sep 15 21:42:20.664484 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.664145    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw" event={"ID":"71381e8f-030b-4482-a8fc-1a1133365237","Type":"ContainerDied","Data":"06a1145e74e0021cf180cfc75ae8513d86d1cf9a4124f1dfd698f2ed33371dd6"}
Sep 15 21:42:20.664484 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.664175    2575 scope.go:117] "RemoveContainer" containerID="c9a830540fdeb5804c2c80915e2669fa982fb599d64ca3fe3610543146fe371a"
Sep 15 21:42:20.666053 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.666029    2575 generic.go:356] "Generic (PLEG): container finished" podID="7880acb1-c1d9-4509-ae56-1cd81a2ca6c8" containerID="773cac1d44fa13f24e02ab7df85e5b924902de47c6c15f565ddab7f58a7ac557" exitCode=0
Sep 15 21:42:20.666191 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.666094    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"
Sep 15 21:42:20.666191 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.666114    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr" event={"ID":"7880acb1-c1d9-4509-ae56-1cd81a2ca6c8","Type":"ContainerDied","Data":"773cac1d44fa13f24e02ab7df85e5b924902de47c6c15f565ddab7f58a7ac557"}
Sep 15 21:42:20.666191 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.666171    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr" event={"ID":"7880acb1-c1d9-4509-ae56-1cd81a2ca6c8","Type":"ContainerDied","Data":"d7235fb7f004abff176e24b9237b195c09767d61ab5eb20029b451d53b8775ae"}
Sep 15 21:42:20.675124 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.675094    2575 scope.go:117] "RemoveContainer" containerID="c9a830540fdeb5804c2c80915e2669fa982fb599d64ca3fe3610543146fe371a"
Sep 15 21:42:20.675557 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:42:20.675537    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a830540fdeb5804c2c80915e2669fa982fb599d64ca3fe3610543146fe371a\": container with ID starting with c9a830540fdeb5804c2c80915e2669fa982fb599d64ca3fe3610543146fe371a not found: ID does not exist" containerID="c9a830540fdeb5804c2c80915e2669fa982fb599d64ca3fe3610543146fe371a"
Sep 15 21:42:20.675634 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.675574    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a830540fdeb5804c2c80915e2669fa982fb599d64ca3fe3610543146fe371a"} err="failed to get container status \"c9a830540fdeb5804c2c80915e2669fa982fb599d64ca3fe3610543146fe371a\": rpc error: code = NotFound desc = could not find container \"c9a830540fdeb5804c2c80915e2669fa982fb599d64ca3fe3610543146fe371a\": container with ID starting with c9a830540fdeb5804c2c80915e2669fa982fb599d64ca3fe3610543146fe371a not found: ID does not exist"
Sep 15 21:42:20.675634 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.675599    2575 scope.go:117] "RemoveContainer" containerID="773cac1d44fa13f24e02ab7df85e5b924902de47c6c15f565ddab7f58a7ac557"
Sep 15 21:42:20.685132 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.685108    2575 scope.go:117] "RemoveContainer" containerID="773cac1d44fa13f24e02ab7df85e5b924902de47c6c15f565ddab7f58a7ac557"
Sep 15 21:42:20.685525 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:42:20.685504    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"773cac1d44fa13f24e02ab7df85e5b924902de47c6c15f565ddab7f58a7ac557\": container with ID starting with 773cac1d44fa13f24e02ab7df85e5b924902de47c6c15f565ddab7f58a7ac557 not found: ID does not exist" containerID="773cac1d44fa13f24e02ab7df85e5b924902de47c6c15f565ddab7f58a7ac557"
Sep 15 21:42:20.685579 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.685535    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"773cac1d44fa13f24e02ab7df85e5b924902de47c6c15f565ddab7f58a7ac557"} err="failed to get container status \"773cac1d44fa13f24e02ab7df85e5b924902de47c6c15f565ddab7f58a7ac557\": rpc error: code = NotFound desc = could not find container \"773cac1d44fa13f24e02ab7df85e5b924902de47c6c15f565ddab7f58a7ac557\": container with ID starting with 773cac1d44fa13f24e02ab7df85e5b924902de47c6c15f565ddab7f58a7ac557 not found: ID does not exist"
Sep 15 21:42:20.690247 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.690206    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"]
Sep 15 21:42:20.692602 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.692566    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5416b7-6585d467bb-x2fdw"]
Sep 15 21:42:20.707116 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.707073    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"]
Sep 15 21:42:20.709163 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.709136    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5416b7-7db9bd998c-wzgsr"]
Sep 15 21:42:20.748315 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.748276    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03ca6cea-ec0c-4d55-8b9f-02c74af881e4" path="/var/lib/kubelet/pods/03ca6cea-ec0c-4d55-8b9f-02c74af881e4/volumes"
Sep 15 21:42:20.748692 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.748676    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71381e8f-030b-4482-a8fc-1a1133365237" path="/var/lib/kubelet/pods/71381e8f-030b-4482-a8fc-1a1133365237/volumes"
Sep 15 21:42:20.749031 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.749018    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7880acb1-c1d9-4509-ae56-1cd81a2ca6c8" path="/var/lib/kubelet/pods/7880acb1-c1d9-4509-ae56-1cd81a2ca6c8/volumes"
Sep 15 21:42:20.749654 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:20.749638    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e03d0f17-0241-41b4-af38-568e50659d5a" path="/var/lib/kubelet/pods/e03d0f17-0241-41b4-af38-568e50659d5a/volumes"
Sep 15 21:42:27.062411 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.062328    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"]
Sep 15 21:42:27.062850 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.062818    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44d66ff9-5e8e-429d-a4db-4edf90c01731" containerName="payload-processing"
Sep 15 21:42:27.062850 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.062832    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="44d66ff9-5e8e-429d-a4db-4edf90c01731" containerName="payload-processing"
Sep 15 21:42:27.062850 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.062845    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03ca6cea-ec0c-4d55-8b9f-02c74af881e4" containerName="istio-proxy"
Sep 15 21:42:27.062850 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.062851    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="03ca6cea-ec0c-4d55-8b9f-02c74af881e4" containerName="istio-proxy"
Sep 15 21:42:27.063041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.062870    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="247b0b8f-aa1f-467a-8fb6-70ff9e7c0091" containerName="payload-pre-processing"
Sep 15 21:42:27.063041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.062879    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="247b0b8f-aa1f-467a-8fb6-70ff9e7c0091" containerName="payload-pre-processing"
Sep 15 21:42:27.063041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.062891    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7880acb1-c1d9-4509-ae56-1cd81a2ca6c8" containerName="payload-processing"
Sep 15 21:42:27.063041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.062896    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7880acb1-c1d9-4509-ae56-1cd81a2ca6c8" containerName="payload-processing"
Sep 15 21:42:27.063041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.062904    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e03d0f17-0241-41b4-af38-568e50659d5a" containerName="revoke-keys"
Sep 15 21:42:27.063041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.062909    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e03d0f17-0241-41b4-af38-568e50659d5a" containerName="revoke-keys"
Sep 15 21:42:27.063041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.062926    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71381e8f-030b-4482-a8fc-1a1133365237" containerName="payload-pre-processing"
Sep 15 21:42:27.063041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.062938    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="71381e8f-030b-4482-a8fc-1a1133365237" containerName="payload-pre-processing"
Sep 15 21:42:27.063041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.063025    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="03ca6cea-ec0c-4d55-8b9f-02c74af881e4" containerName="istio-proxy"
Sep 15 21:42:27.063041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.063036    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="247b0b8f-aa1f-467a-8fb6-70ff9e7c0091" containerName="payload-pre-processing"
Sep 15 21:42:27.063041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.063043    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="71381e8f-030b-4482-a8fc-1a1133365237" containerName="payload-pre-processing"
Sep 15 21:42:27.063377 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.063049    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e03d0f17-0241-41b4-af38-568e50659d5a" containerName="revoke-keys"
Sep 15 21:42:27.063377 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.063057    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="44d66ff9-5e8e-429d-a4db-4edf90c01731" containerName="payload-processing"
Sep 15 21:42:27.063377 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.063062    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7880acb1-c1d9-4509-ae56-1cd81a2ca6c8" containerName="payload-processing"
Sep 15 21:42:27.067975 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.067949    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.070333 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.070304    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 21:42:27.070491 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.070377    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-4a3a26-openshift-default-dockercfg-rmhcj\""
Sep 15 21:42:27.081027 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.080994    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"]
Sep 15 21:42:27.136854 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.136812    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-workload-socket\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.137147 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.136875    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-data\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.137147 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.136963    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istiod-ca-cert\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.137147 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.137064    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-workload-certs\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.137147 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.137111    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-envoy\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.137395 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.137231    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-podinfo\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.137395 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.137332    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp896\" (UniqueName: \"kubernetes.io/projected/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-kube-api-access-mp896\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.137395 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.137381    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-credential-socket\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.137550 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.137428    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-token\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.238839 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.238748    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-credential-socket\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.238839 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.238852    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-token\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.239125 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.238880    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-workload-socket\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.239125 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.238905    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-data\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.239125 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.238928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istiod-ca-cert\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.239125 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.239005    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-workload-certs\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.239125 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.239040    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-envoy\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.239125 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.239106    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-podinfo\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.239406 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.239229    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-credential-socket\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.239406 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.239231    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp896\" (UniqueName: \"kubernetes.io/projected/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-kube-api-access-mp896\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.239406 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.239288    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-workload-socket\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.239406 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.239338    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-data\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.239533 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.239492    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-workload-certs\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.239714 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.239695    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istiod-ca-cert\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.241787 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.241740    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-envoy\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.241914 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.241840    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-podinfo\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.253245 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.253212    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-token\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.256287 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.256257    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp896\" (UniqueName: \"kubernetes.io/projected/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-kube-api-access-mp896\") pod \"e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") " pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.380856 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.380812    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:27.531872 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.531827    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:42:27.663745 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.663717    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"]
Sep 15 21:42:27.665681 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:42:27.665654    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7cd896_0afa_48d3_a34e_a74f7b3342f9.slice/crio-f64b15254af6256ce0bfe25300a794f3b55cf67140be7eb5eebd427136c94f00 WatchSource:0}: Error finding container f64b15254af6256ce0bfe25300a794f3b55cf67140be7eb5eebd427136c94f00: Status 404 returned error can't find the container with id f64b15254af6256ce0bfe25300a794f3b55cf67140be7eb5eebd427136c94f00
Sep 15 21:42:27.668071 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.668037    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:42:27.668138 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.668113    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:42:27.668175 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.668144    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:42:27.711613 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:27.711574    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8" event={"ID":"ae7cd896-0afa-48d3-a34e-a74f7b3342f9","Type":"ContainerStarted","Data":"f64b15254af6256ce0bfe25300a794f3b55cf67140be7eb5eebd427136c94f00"}
Sep 15 21:42:28.719004 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:28.718968    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8" event={"ID":"ae7cd896-0afa-48d3-a34e-a74f7b3342f9","Type":"ContainerStarted","Data":"eaf16962641130eed0135d23ea5cd40123c1cb5bbb0c8423017a5402cb8a3f23"}
Sep 15 21:42:28.739435 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:28.739357    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8" podStartSLOduration=1.739334494 podStartE2EDuration="1.739334494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:42:28.736340884 +0000 UTC m=+986.548349327" watchObservedRunningTime="2026-09-15 21:42:28.739334494 +0000 UTC m=+986.551343037"
Sep 15 21:42:29.381834 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:29.381787    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:29.387898 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:29.387864    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:29.725242 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:29.725153    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:29.726465 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:29.726445    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:34.220547 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.220505    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"]
Sep 15 21:42:34.221048 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.221029    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03ca6cea-ec0c-4d55-8b9f-02c74af881e4" containerName="istio-proxy"
Sep 15 21:42:34.221108 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.221051    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="03ca6cea-ec0c-4d55-8b9f-02c74af881e4" containerName="istio-proxy"
Sep 15 21:42:34.221185 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.221175    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="03ca6cea-ec0c-4d55-8b9f-02c74af881e4" containerName="istio-proxy"
Sep 15 21:42:34.224644 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.224618    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"
Sep 15 21:42:34.227225 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.227198    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-4a3a26\""
Sep 15 21:42:34.227408 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.227198    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-4a3a26-dockercfg-q284z\""
Sep 15 21:42:34.234217 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.234183    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"]
Sep 15 21:42:34.257872 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.257833    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"]
Sep 15 21:42:34.262621 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.262576    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"]
Sep 15 21:42:34.262869 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.262685    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"
Sep 15 21:42:34.316335 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.316299    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh\" (UID: \"ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"
Sep 15 21:42:34.316543 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.316494    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmz7v\" (UniqueName: \"kubernetes.io/projected/ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73-kube-api-access-xmz7v\") pod \"payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh\" (UID: \"ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"
Sep 15 21:42:34.417557 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.417518    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmz7v\" (UniqueName: \"kubernetes.io/projected/ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73-kube-api-access-xmz7v\") pod \"payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh\" (UID: \"ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"
Sep 15 21:42:34.417787 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.417599    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh\" (UID: \"ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"
Sep 15 21:42:34.417787 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.417681    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vlxc\" (UniqueName: \"kubernetes.io/projected/39a528d4-d94b-4911-ae0b-4dc223aab4d0-kube-api-access-8vlxc\") pod \"payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8\" (UID: \"39a528d4-d94b-4911-ae0b-4dc223aab4d0\") " pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"
Sep 15 21:42:34.417926 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.417812    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a528d4-d94b-4911-ae0b-4dc223aab4d0-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8\" (UID: \"39a528d4-d94b-4911-ae0b-4dc223aab4d0\") " pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"
Sep 15 21:42:34.418675 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.418643    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh\" (UID: \"ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"
Sep 15 21:42:34.426567 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.426533    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmz7v\" (UniqueName: \"kubernetes.io/projected/ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73-kube-api-access-xmz7v\") pod \"payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh\" (UID: \"ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"
Sep 15 21:42:34.518868 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.518723    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vlxc\" (UniqueName: \"kubernetes.io/projected/39a528d4-d94b-4911-ae0b-4dc223aab4d0-kube-api-access-8vlxc\") pod \"payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8\" (UID: \"39a528d4-d94b-4911-ae0b-4dc223aab4d0\") " pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"
Sep 15 21:42:34.518868 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.518835    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a528d4-d94b-4911-ae0b-4dc223aab4d0-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8\" (UID: \"39a528d4-d94b-4911-ae0b-4dc223aab4d0\") " pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"
Sep 15 21:42:34.519543 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.519523    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a528d4-d94b-4911-ae0b-4dc223aab4d0-plugins-config-volume\") pod \"payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8\" (UID: \"39a528d4-d94b-4911-ae0b-4dc223aab4d0\") " pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"
Sep 15 21:42:34.527009 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.526978    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vlxc\" (UniqueName: \"kubernetes.io/projected/39a528d4-d94b-4911-ae0b-4dc223aab4d0-kube-api-access-8vlxc\") pod \"payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8\" (UID: \"39a528d4-d94b-4911-ae0b-4dc223aab4d0\") " pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"
Sep 15 21:42:34.537937 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.537907    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"
Sep 15 21:42:34.575122 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.575081    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"
Sep 15 21:42:34.821229 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.821199    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"]
Sep 15 21:42:34.823114 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:42:34.823083    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce48a18d_f3a5_4df4_a5db_f2d1a5d9cb73.slice/crio-36be0e23a040b0ddf1a7838367e70af7803c11cdaa24c7babba8d0dfc129eed9 WatchSource:0}: Error finding container 36be0e23a040b0ddf1a7838367e70af7803c11cdaa24c7babba8d0dfc129eed9: Status 404 returned error can't find the container with id 36be0e23a040b0ddf1a7838367e70af7803c11cdaa24c7babba8d0dfc129eed9
Sep 15 21:42:34.864267 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:34.864237    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"]
Sep 15 21:42:34.866042 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:42:34.866012    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a528d4_d94b_4911_ae0b_4dc223aab4d0.slice/crio-3077864b19363fe8cc6da3448bcaf4178bf088013a4825b69cf5e7e0f093b835 WatchSource:0}: Error finding container 3077864b19363fe8cc6da3448bcaf4178bf088013a4825b69cf5e7e0f093b835: Status 404 returned error can't find the container with id 3077864b19363fe8cc6da3448bcaf4178bf088013a4825b69cf5e7e0f093b835
Sep 15 21:42:35.768245 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:35.768195    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8" event={"ID":"39a528d4-d94b-4911-ae0b-4dc223aab4d0","Type":"ContainerStarted","Data":"187e081c4efa3b8ac5877998c18340c8c7bb94aa294f2abf742925dd5e59cffc"}
Sep 15 21:42:35.768726 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:35.768313    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8" event={"ID":"39a528d4-d94b-4911-ae0b-4dc223aab4d0","Type":"ContainerStarted","Data":"3077864b19363fe8cc6da3448bcaf4178bf088013a4825b69cf5e7e0f093b835"}
Sep 15 21:42:35.768726 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:35.768344    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"
Sep 15 21:42:35.770061 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:35.770029    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh" event={"ID":"ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73","Type":"ContainerStarted","Data":"06fc0051848e1c720153f5c4703f91025b27baefe508f2dcd69bc7769cb6824a"}
Sep 15 21:42:35.770195 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:35.770083    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh" event={"ID":"ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73","Type":"ContainerStarted","Data":"36be0e23a040b0ddf1a7838367e70af7803c11cdaa24c7babba8d0dfc129eed9"}
Sep 15 21:42:35.770195 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:35.770110    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"
Sep 15 21:42:35.783559 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:35.783487    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8" podStartSLOduration=1.7834670959999999 podStartE2EDuration="1.783467096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:42:35.78292465 +0000 UTC m=+993.594933096" watchObservedRunningTime="2026-09-15 21:42:35.783467096 +0000 UTC m=+993.595475535"
Sep 15 21:42:35.800203 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:35.800144    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh" podStartSLOduration=1.80012616 podStartE2EDuration="1.80012616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:42:35.799308782 +0000 UTC m=+993.611317228" watchObservedRunningTime="2026-09-15 21:42:35.80012616 +0000 UTC m=+993.612134604"
Sep 15 21:42:40.239444 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:40.239398    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:42:46.778701 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:46.778665    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"
Sep 15 21:42:46.779261 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:46.778788    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"
Sep 15 21:42:51.868854 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:51.868812    2575 generic.go:356] "Generic (PLEG): container finished" podID="ae7cd896-0afa-48d3-a34e-a74f7b3342f9" containerID="eaf16962641130eed0135d23ea5cd40123c1cb5bbb0c8423017a5402cb8a3f23" exitCode=0
Sep 15 21:42:51.869359 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:51.868890    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8" event={"ID":"ae7cd896-0afa-48d3-a34e-a74f7b3342f9","Type":"ContainerDied","Data":"eaf16962641130eed0135d23ea5cd40123c1cb5bbb0c8423017a5402cb8a3f23"}
Sep 15 21:42:51.869359 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:51.869348    2575 scope.go:117] "RemoveContainer" containerID="eaf16962641130eed0135d23ea5cd40123c1cb5bbb0c8423017a5402cb8a3f23"
Sep 15 21:42:51.870631 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:51.870592    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:42:51.870745 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:51.870694    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:42:51.870839 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:51.870744    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:42:52.885505 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:52.885463    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8" event={"ID":"ae7cd896-0afa-48d3-a34e-a74f7b3342f9","Type":"ContainerStarted","Data":"0a49629f0d1d79e0e71627d565141bb28f5764e3499d5c1912e14ef4e61f9a8b"}
Sep 15 21:42:53.381066 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:53.381019    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:53.381464 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:53.381444    2575 patch_prober.go:28] interesting pod/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 15 21:42:53.381516 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:53.381497    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8" podUID="ae7cd896-0afa-48d3-a34e-a74f7b3342f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 15 21:42:54.381541 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:54.381488    2575 patch_prober.go:28] interesting pod/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 15 21:42:54.382033 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:54.381559    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8" podUID="ae7cd896-0afa-48d3-a34e-a74f7b3342f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 15 21:42:55.382016 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:55.381971    2575 patch_prober.go:28] interesting pod/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 15 21:42:55.382442 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:55.382034    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8" podUID="ae7cd896-0afa-48d3-a34e-a74f7b3342f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 15 21:42:56.381726 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:56.381671    2575 patch_prober.go:28] interesting pod/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 15 21:42:56.382030 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:56.381748    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8" podUID="ae7cd896-0afa-48d3-a34e-a74f7b3342f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 15 21:42:57.382215 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:57.382173    2575 patch_prober.go:28] interesting pod/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 15 21:42:57.382663 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:57.382234    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8" podUID="ae7cd896-0afa-48d3-a34e-a74f7b3342f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 15 21:42:58.381822 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:58.381291    2575 patch_prober.go:28] interesting pod/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 15 21:42:58.381822 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:58.381371    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8" podUID="ae7cd896-0afa-48d3-a34e-a74f7b3342f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 15 21:42:59.386717 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:59.386683    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:59.387140 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:59.387099    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:42:59.388000 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:42:59.387979    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:43:12.338490 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:12.338449    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:43:29.742862 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:29.742821    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:43:47.826276 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:47.826237    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"]
Sep 15 21:43:47.830214 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:47.830185    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:47.834939 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:47.834259    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4a3a26\"/\"explicit-model-4a3a26-kserve-self-signed-certs\""
Sep 15 21:43:47.834939 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:47.834299    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4a3a26\"/\"openshift-service-ca.crt\""
Sep 15 21:43:47.834939 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:47.833597    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-4a3a26\"/\"default-dockercfg-cfntp\""
Sep 15 21:43:47.836498 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:47.835529    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-4a3a26\"/\"kube-root-ca.crt\""
Sep 15 21:43:47.840586 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:47.840550    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"]
Sep 15 21:43:47.927418 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:47.927371    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6pgr\" (UniqueName: \"kubernetes.io/projected/1f52a028-b94f-4429-940c-1b66cee6c421-kube-api-access-p6pgr\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:47.927635 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:47.927448    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f52a028-b94f-4429-940c-1b66cee6c421-tls-certs\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:47.927635 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:47.927508    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-dshm\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:47.927635 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:47.927558    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-tmp-dir\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:47.927813 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:47.927666    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-model-cache\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:47.927813 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:47.927734    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-home\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:48.028574 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:48.028529    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f52a028-b94f-4429-940c-1b66cee6c421-tls-certs\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:48.028574 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:48.028578    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-dshm\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:48.028924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:48.028611    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-tmp-dir\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:48.028924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:48.028695    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-model-cache\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:48.028924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:48.028798    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-home\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:48.028924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:48.028861    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6pgr\" (UniqueName: \"kubernetes.io/projected/1f52a028-b94f-4429-940c-1b66cee6c421-kube-api-access-p6pgr\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:48.029175 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:48.029148    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-tmp-dir\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:48.029218 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:48.029169    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-model-cache\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:48.029218 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:48.029206    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-home\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:48.031289 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:48.031260    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-dshm\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:48.031476 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:48.031454    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f52a028-b94f-4429-940c-1b66cee6c421-tls-certs\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:48.038216 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:48.038184    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6pgr\" (UniqueName: \"kubernetes.io/projected/1f52a028-b94f-4429-940c-1b66cee6c421-kube-api-access-p6pgr\") pod \"explicit-model-4a3a26-kserve-547bfdc59f-xs2dl\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") " pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:48.153923 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:48.153879    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:48.431896 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:48.431745    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"]
Sep 15 21:43:48.434581 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:43:48.434553    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f52a028_b94f_4429_940c_1b66cee6c421.slice/crio-1c5419b51d8d62cd6c3e546f0a9df2ad46f95d3a64bd0ecf8c57e43f5e6e45dd WatchSource:0}: Error finding container 1c5419b51d8d62cd6c3e546f0a9df2ad46f95d3a64bd0ecf8c57e43f5e6e45dd: Status 404 returned error can't find the container with id 1c5419b51d8d62cd6c3e546f0a9df2ad46f95d3a64bd0ecf8c57e43f5e6e45dd
Sep 15 21:43:49.218472 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:49.218429    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl" event={"ID":"1f52a028-b94f-4429-940c-1b66cee6c421","Type":"ContainerStarted","Data":"6fa55a56b3aa0ca96064867c60644c7b0d160c99c1f84f9074f8cb362d079329"}
Sep 15 21:43:49.218472 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:49.218474    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl" event={"ID":"1f52a028-b94f-4429-940c-1b66cee6c421","Type":"ContainerStarted","Data":"1c5419b51d8d62cd6c3e546f0a9df2ad46f95d3a64bd0ecf8c57e43f5e6e45dd"}
Sep 15 21:43:49.234632 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:49.234569    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl" podStartSLOduration=2.234551746 podStartE2EDuration="2.234551746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:43:49.233429859 +0000 UTC m=+1067.045438328" watchObservedRunningTime="2026-09-15 21:43:49.234551746 +0000 UTC m=+1067.046560190"
Sep 15 21:43:58.154955 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:58.154847    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:58.154955 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:58.154926    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:58.168151 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:58.168115    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:58.287917 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:58.287876    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:43:59.653167 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:43:59.653128    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:44:00.290020 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.289979    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"]
Sep 15 21:44:00.294072 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.294042    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"
Sep 15 21:44:00.296539 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.296510    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e2d4753a\""
Sep 15 21:44:00.304536 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.304510    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"]
Sep 15 21:44:00.321247 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.321209    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"]
Sep 15 21:44:00.325111 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.325082    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"
Sep 15 21:44:00.327643 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.327616    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e2d4753a-dockercfg-xs8ss\""
Sep 15 21:44:00.333481 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.333450    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"]
Sep 15 21:44:00.369964 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.369924    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4lq5\" (UniqueName: \"kubernetes.io/projected/bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd-kube-api-access-w4lq5\") pod \"payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr\" (UID: \"bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"
Sep 15 21:44:00.370156 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.370093    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr\" (UID: \"bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"
Sep 15 21:44:00.471101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.471062    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27af269b-9953-4ed3-b9e4-d35fb1840e43-plugins-config-volume\") pod \"payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq\" (UID: \"27af269b-9953-4ed3-b9e4-d35fb1840e43\") " pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"
Sep 15 21:44:00.471301 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.471136    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr\" (UID: \"bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"
Sep 15 21:44:00.471301 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.471259    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chl7n\" (UniqueName: \"kubernetes.io/projected/27af269b-9953-4ed3-b9e4-d35fb1840e43-kube-api-access-chl7n\") pod \"payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq\" (UID: \"27af269b-9953-4ed3-b9e4-d35fb1840e43\") " pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"
Sep 15 21:44:00.471382 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.471335    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4lq5\" (UniqueName: \"kubernetes.io/projected/bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd-kube-api-access-w4lq5\") pod \"payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr\" (UID: \"bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"
Sep 15 21:44:00.471973 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.471953    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr\" (UID: \"bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"
Sep 15 21:44:00.479733 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.479703    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4lq5\" (UniqueName: \"kubernetes.io/projected/bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd-kube-api-access-w4lq5\") pod \"payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr\" (UID: \"bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"
Sep 15 21:44:00.572428 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.572321    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chl7n\" (UniqueName: \"kubernetes.io/projected/27af269b-9953-4ed3-b9e4-d35fb1840e43-kube-api-access-chl7n\") pod \"payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq\" (UID: \"27af269b-9953-4ed3-b9e4-d35fb1840e43\") " pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"
Sep 15 21:44:00.572428 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.572421    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27af269b-9953-4ed3-b9e4-d35fb1840e43-plugins-config-volume\") pod \"payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq\" (UID: \"27af269b-9953-4ed3-b9e4-d35fb1840e43\") " pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"
Sep 15 21:44:00.573207 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.573186    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27af269b-9953-4ed3-b9e4-d35fb1840e43-plugins-config-volume\") pod \"payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq\" (UID: \"27af269b-9953-4ed3-b9e4-d35fb1840e43\") " pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"
Sep 15 21:44:00.580314 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.580282    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chl7n\" (UniqueName: \"kubernetes.io/projected/27af269b-9953-4ed3-b9e4-d35fb1840e43-kube-api-access-chl7n\") pod \"payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq\" (UID: \"27af269b-9953-4ed3-b9e4-d35fb1840e43\") " pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"
Sep 15 21:44:00.607123 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.607079    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"
Sep 15 21:44:00.638873 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.638836    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"
Sep 15 21:44:00.896889 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.896856    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"]
Sep 15 21:44:00.898356 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:44:00.898323    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc8f9c3_88bc_4e7e_b40f_57bc86d1dfdd.slice/crio-b405215f8199a55c8cc07989f8ac40d9304883cbc227562387100ac61c6a44a1 WatchSource:0}: Error finding container b405215f8199a55c8cc07989f8ac40d9304883cbc227562387100ac61c6a44a1: Status 404 returned error can't find the container with id b405215f8199a55c8cc07989f8ac40d9304883cbc227562387100ac61c6a44a1
Sep 15 21:44:00.943271 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:00.943248    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"]
Sep 15 21:44:00.945853 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:44:00.945824    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27af269b_9953_4ed3_b9e4_d35fb1840e43.slice/crio-7d4600c32dd6d57df206c5b63b87eb4fe6f8a06b304d59c9b03bc7aba21d0dd0 WatchSource:0}: Error finding container 7d4600c32dd6d57df206c5b63b87eb4fe6f8a06b304d59c9b03bc7aba21d0dd0: Status 404 returned error can't find the container with id 7d4600c32dd6d57df206c5b63b87eb4fe6f8a06b304d59c9b03bc7aba21d0dd0
Sep 15 21:44:01.296831 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:01.296699    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr" event={"ID":"bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd","Type":"ContainerStarted","Data":"2efafc58d46ca5a57f58c4f64498635765586628d1d92c5aa81b9d2ead3aa737"}
Sep 15 21:44:01.296831 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:01.296799    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr" event={"ID":"bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd","Type":"ContainerStarted","Data":"b405215f8199a55c8cc07989f8ac40d9304883cbc227562387100ac61c6a44a1"}
Sep 15 21:44:01.297084 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:01.296847    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"
Sep 15 21:44:01.298916 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:01.298886    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq" event={"ID":"27af269b-9953-4ed3-b9e4-d35fb1840e43","Type":"ContainerStarted","Data":"1c68dd836cffff77a11096917b46f02819f8e37276d6a2a29450967a90bc6a3d"}
Sep 15 21:44:01.299102 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:01.298943    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"
Sep 15 21:44:01.299102 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:01.298967    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq" event={"ID":"27af269b-9953-4ed3-b9e4-d35fb1840e43","Type":"ContainerStarted","Data":"7d4600c32dd6d57df206c5b63b87eb4fe6f8a06b304d59c9b03bc7aba21d0dd0"}
Sep 15 21:44:01.313475 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:01.313418    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr" podStartSLOduration=1.313400286 podStartE2EDuration="1.313400286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:44:01.311260759 +0000 UTC m=+1079.123269207" watchObservedRunningTime="2026-09-15 21:44:01.313400286 +0000 UTC m=+1079.125408726"
Sep 15 21:44:01.328134 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:01.328071    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq" podStartSLOduration=1.3280516470000001 podStartE2EDuration="1.328051647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:44:01.327109044 +0000 UTC m=+1079.139117486" watchObservedRunningTime="2026-09-15 21:44:01.328051647 +0000 UTC m=+1079.140060090"
Sep 15 21:44:12.307000 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:12.306950    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"
Sep 15 21:44:12.307420 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:12.307202    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"
Sep 15 21:44:23.014789 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:23.014726    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7"]
Sep 15 21:44:23.020092 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:23.020064    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7"
Sep 15 21:44:23.023047 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:23.023013    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:44:23.023232 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:23.023171    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 21:44:23.026094 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:23.026056    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7"]
Sep 15 21:44:23.123952 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:23.123905    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7\" (UID: \"d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7"
Sep 15 21:44:23.225692 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:23.225647    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7\" (UID: \"d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7"
Sep 15 21:44:23.226356 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:23.226333    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7\" (UID: \"d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7"
Sep 15 21:44:23.333971 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:23.333858    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7"
Sep 15 21:44:23.614915 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:23.614884    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7"]
Sep 15 21:44:23.617248 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:44:23.617216    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f2fda3_c5e9_4c29_9ee3_e1805e2a55c0.slice/crio-f035afd38d1f83e0e0046b86410398cb47146a4325e11e47514387deb598e797 WatchSource:0}: Error finding container f035afd38d1f83e0e0046b86410398cb47146a4325e11e47514387deb598e797: Status 404 returned error can't find the container with id f035afd38d1f83e0e0046b86410398cb47146a4325e11e47514387deb598e797
Sep 15 21:44:24.449144 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:24.449109    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7" event={"ID":"d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0","Type":"ContainerStarted","Data":"ad1282ae4100e8a242c4298007e34ee8e0279ca33143d961089dc440d460135f"}
Sep 15 21:44:24.449586 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:24.449163    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7" event={"ID":"d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0","Type":"ContainerStarted","Data":"f035afd38d1f83e0e0046b86410398cb47146a4325e11e47514387deb598e797"}
Sep 15 21:44:24.466713 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:24.466655    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7" podStartSLOduration=2.466633571 podStartE2EDuration="2.466633571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:44:24.463972083 +0000 UTC m=+1102.275980531" watchObservedRunningTime="2026-09-15 21:44:24.466633571 +0000 UTC m=+1102.278642091"
Sep 15 21:44:25.456724 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:25.456687    2575 generic.go:356] "Generic (PLEG): container finished" podID="d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0" containerID="ad1282ae4100e8a242c4298007e34ee8e0279ca33143d961089dc440d460135f" exitCode=0
Sep 15 21:44:25.457238 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:25.456795    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7" event={"ID":"d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0","Type":"ContainerDied","Data":"ad1282ae4100e8a242c4298007e34ee8e0279ca33143d961089dc440d460135f"}
Sep 15 21:44:26.611255 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:26.611223    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7"
Sep 15 21:44:26.662500 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:26.662466    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0-maas-api-service-ca\") pod \"d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0\" (UID: \"d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0\") "
Sep 15 21:44:26.662942 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:26.662917    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0" (UID: "d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:44:26.764232 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:26.764188    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:44:27.470531 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:27.470496    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7"
Sep 15 21:44:27.470531 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:27.470510    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7" event={"ID":"d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0","Type":"ContainerDied","Data":"f035afd38d1f83e0e0046b86410398cb47146a4325e11e47514387deb598e797"}
Sep 15 21:44:27.470864 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:27.470561    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f035afd38d1f83e0e0046b86410398cb47146a4325e11e47514387deb598e797"
Sep 15 21:44:27.936423 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:27.936369    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:44:38.266868 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.266819    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"]
Sep 15 21:44:38.267526 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.267123    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8" podUID="39a528d4-d94b-4911-ae0b-4dc223aab4d0" containerName="payload-processing" containerID="cri-o://187e081c4efa3b8ac5877998c18340c8c7bb94aa294f2abf742925dd5e59cffc" gracePeriod=30
Sep 15 21:44:38.274975 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.274934    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"]
Sep 15 21:44:38.275221 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.275195    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh" podUID="ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73" containerName="payload-pre-processing" containerID="cri-o://06fc0051848e1c720153f5c4703f91025b27baefe508f2dcd69bc7769cb6824a" gracePeriod=30
Sep 15 21:44:38.429993 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.429950    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7"]
Sep 15 21:44:38.432341 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.432299    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-4a3a26-bc4m7"]
Sep 15 21:44:38.545379 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.545343    2575 generic.go:356] "Generic (PLEG): container finished" podID="39a528d4-d94b-4911-ae0b-4dc223aab4d0" containerID="187e081c4efa3b8ac5877998c18340c8c7bb94aa294f2abf742925dd5e59cffc" exitCode=0
Sep 15 21:44:38.545559 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.545413    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8" event={"ID":"39a528d4-d94b-4911-ae0b-4dc223aab4d0","Type":"ContainerDied","Data":"187e081c4efa3b8ac5877998c18340c8c7bb94aa294f2abf742925dd5e59cffc"}
Sep 15 21:44:38.547228 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.547197    2575 generic.go:356] "Generic (PLEG): container finished" podID="ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73" containerID="06fc0051848e1c720153f5c4703f91025b27baefe508f2dcd69bc7769cb6824a" exitCode=0
Sep 15 21:44:38.547364 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.547269    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh" event={"ID":"ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73","Type":"ContainerDied","Data":"06fc0051848e1c720153f5c4703f91025b27baefe508f2dcd69bc7769cb6824a"}
Sep 15 21:44:38.572522 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.572496    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"
Sep 15 21:44:38.576079 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.576051    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"
Sep 15 21:44:38.698369 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.698327    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a528d4-d94b-4911-ae0b-4dc223aab4d0-plugins-config-volume\") pod \"39a528d4-d94b-4911-ae0b-4dc223aab4d0\" (UID: \"39a528d4-d94b-4911-ae0b-4dc223aab4d0\") "
Sep 15 21:44:38.698589 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.698395    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73-plugins-config-volume\") pod \"ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73\" (UID: \"ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73\") "
Sep 15 21:44:38.698589 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.698416    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmz7v\" (UniqueName: \"kubernetes.io/projected/ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73-kube-api-access-xmz7v\") pod \"ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73\" (UID: \"ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73\") "
Sep 15 21:44:38.698589 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.698447    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vlxc\" (UniqueName: \"kubernetes.io/projected/39a528d4-d94b-4911-ae0b-4dc223aab4d0-kube-api-access-8vlxc\") pod \"39a528d4-d94b-4911-ae0b-4dc223aab4d0\" (UID: \"39a528d4-d94b-4911-ae0b-4dc223aab4d0\") "
Sep 15 21:44:38.698889 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.698863    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a528d4-d94b-4911-ae0b-4dc223aab4d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39a528d4-d94b-4911-ae0b-4dc223aab4d0" (UID: "39a528d4-d94b-4911-ae0b-4dc223aab4d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:44:38.698955 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.698863    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73" (UID: "ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:44:38.700920 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.700880    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73-kube-api-access-xmz7v" (OuterVolumeSpecName: "kube-api-access-xmz7v") pod "ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73" (UID: "ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73"). InnerVolumeSpecName "kube-api-access-xmz7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:38.701040 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.700952    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a528d4-d94b-4911-ae0b-4dc223aab4d0-kube-api-access-8vlxc" (OuterVolumeSpecName: "kube-api-access-8vlxc") pod "39a528d4-d94b-4911-ae0b-4dc223aab4d0" (UID: "39a528d4-d94b-4911-ae0b-4dc223aab4d0"). InnerVolumeSpecName "kube-api-access-8vlxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:38.749082 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.749042    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0" path="/var/lib/kubelet/pods/d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0/volumes"
Sep 15 21:44:38.800170 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.800063    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a528d4-d94b-4911-ae0b-4dc223aab4d0-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:44:38.800170 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.800099    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:44:38.800170 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.800113    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmz7v\" (UniqueName: \"kubernetes.io/projected/ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73-kube-api-access-xmz7v\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:44:38.800170 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:38.800123    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vlxc\" (UniqueName: \"kubernetes.io/projected/39a528d4-d94b-4911-ae0b-4dc223aab4d0-kube-api-access-8vlxc\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:44:39.555455 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:39.555417    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"
Sep 15 21:44:39.555972 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:39.555420    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8" event={"ID":"39a528d4-d94b-4911-ae0b-4dc223aab4d0","Type":"ContainerDied","Data":"3077864b19363fe8cc6da3448bcaf4178bf088013a4825b69cf5e7e0f093b835"}
Sep 15 21:44:39.555972 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:39.555576    2575 scope.go:117] "RemoveContainer" containerID="187e081c4efa3b8ac5877998c18340c8c7bb94aa294f2abf742925dd5e59cffc"
Sep 15 21:44:39.557279 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:39.557123    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"
Sep 15 21:44:39.557279 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:39.557141    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh" event={"ID":"ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73","Type":"ContainerDied","Data":"36be0e23a040b0ddf1a7838367e70af7803c11cdaa24c7babba8d0dfc129eed9"}
Sep 15 21:44:39.566189 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:39.566169    2575 scope.go:117] "RemoveContainer" containerID="06fc0051848e1c720153f5c4703f91025b27baefe508f2dcd69bc7769cb6824a"
Sep 15 21:44:39.574577 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:39.574538    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"]
Sep 15 21:44:39.579680 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:39.579645    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-4a3a26-5898f649f8-x6pgh"]
Sep 15 21:44:39.590371 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:39.590339    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"]
Sep 15 21:44:39.595945 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:39.595912    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-4a3a26-656c86f47c-t7pr8"]
Sep 15 21:44:40.748068 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:40.748033    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39a528d4-d94b-4911-ae0b-4dc223aab4d0" path="/var/lib/kubelet/pods/39a528d4-d94b-4911-ae0b-4dc223aab4d0/volumes"
Sep 15 21:44:40.748468 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:40.748432    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73" path="/var/lib/kubelet/pods/ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73/volumes"
Sep 15 21:44:43.766339 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:43.766301    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"]
Sep 15 21:44:44.131864 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.131821    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"]
Sep 15 21:44:44.132181 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.132124    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq" podUID="27af269b-9953-4ed3-b9e4-d35fb1840e43" containerName="payload-processing" containerID="cri-o://1c68dd836cffff77a11096917b46f02819f8e37276d6a2a29450967a90bc6a3d" gracePeriod=30
Sep 15 21:44:44.137472 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.137440    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"]
Sep 15 21:44:44.137793 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.137731    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr" podUID="bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd" containerName="payload-pre-processing" containerID="cri-o://2efafc58d46ca5a57f58c4f64498635765586628d1d92c5aa81b9d2ead3aa737" gracePeriod=30
Sep 15 21:44:44.401664 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.401570    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"
Sep 15 21:44:44.432615 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.432580    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"
Sep 15 21:44:44.561113 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.561073    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd-plugins-config-volume\") pod \"bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd\" (UID: \"bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd\") "
Sep 15 21:44:44.561113 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.561115    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27af269b-9953-4ed3-b9e4-d35fb1840e43-plugins-config-volume\") pod \"27af269b-9953-4ed3-b9e4-d35fb1840e43\" (UID: \"27af269b-9953-4ed3-b9e4-d35fb1840e43\") "
Sep 15 21:44:44.561348 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.561239    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4lq5\" (UniqueName: \"kubernetes.io/projected/bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd-kube-api-access-w4lq5\") pod \"bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd\" (UID: \"bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd\") "
Sep 15 21:44:44.561348 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.561297    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chl7n\" (UniqueName: \"kubernetes.io/projected/27af269b-9953-4ed3-b9e4-d35fb1840e43-kube-api-access-chl7n\") pod \"27af269b-9953-4ed3-b9e4-d35fb1840e43\" (UID: \"27af269b-9953-4ed3-b9e4-d35fb1840e43\") "
Sep 15 21:44:44.561552 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.561531    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd" (UID: "bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:44:44.561599 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.561546    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27af269b-9953-4ed3-b9e4-d35fb1840e43-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27af269b-9953-4ed3-b9e4-d35fb1840e43" (UID: "27af269b-9953-4ed3-b9e4-d35fb1840e43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:44:44.561683 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.561669    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:44:44.561717 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.561688    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27af269b-9953-4ed3-b9e4-d35fb1840e43-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:44:44.563650 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.563621    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd-kube-api-access-w4lq5" (OuterVolumeSpecName: "kube-api-access-w4lq5") pod "bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd" (UID: "bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd"). InnerVolumeSpecName "kube-api-access-w4lq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:44.563650 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.563631    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27af269b-9953-4ed3-b9e4-d35fb1840e43-kube-api-access-chl7n" (OuterVolumeSpecName: "kube-api-access-chl7n") pod "27af269b-9953-4ed3-b9e4-d35fb1840e43" (UID: "27af269b-9953-4ed3-b9e4-d35fb1840e43"). InnerVolumeSpecName "kube-api-access-chl7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:44.590427 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.590364    2575 generic.go:356] "Generic (PLEG): container finished" podID="bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd" containerID="2efafc58d46ca5a57f58c4f64498635765586628d1d92c5aa81b9d2ead3aa737" exitCode=0
Sep 15 21:44:44.590645 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.590519    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"
Sep 15 21:44:44.590645 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.590523    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr" event={"ID":"bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd","Type":"ContainerDied","Data":"2efafc58d46ca5a57f58c4f64498635765586628d1d92c5aa81b9d2ead3aa737"}
Sep 15 21:44:44.590645 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.590601    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr" event={"ID":"bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd","Type":"ContainerDied","Data":"b405215f8199a55c8cc07989f8ac40d9304883cbc227562387100ac61c6a44a1"}
Sep 15 21:44:44.590645 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.590633    2575 scope.go:117] "RemoveContainer" containerID="2efafc58d46ca5a57f58c4f64498635765586628d1d92c5aa81b9d2ead3aa737"
Sep 15 21:44:44.592607 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.592576    2575 generic.go:356] "Generic (PLEG): container finished" podID="27af269b-9953-4ed3-b9e4-d35fb1840e43" containerID="1c68dd836cffff77a11096917b46f02819f8e37276d6a2a29450967a90bc6a3d" exitCode=0
Sep 15 21:44:44.592736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.592624    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"
Sep 15 21:44:44.592736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.592654    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq" event={"ID":"27af269b-9953-4ed3-b9e4-d35fb1840e43","Type":"ContainerDied","Data":"1c68dd836cffff77a11096917b46f02819f8e37276d6a2a29450967a90bc6a3d"}
Sep 15 21:44:44.592736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.592704    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq" event={"ID":"27af269b-9953-4ed3-b9e4-d35fb1840e43","Type":"ContainerDied","Data":"7d4600c32dd6d57df206c5b63b87eb4fe6f8a06b304d59c9b03bc7aba21d0dd0"}
Sep 15 21:44:44.601558 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.601533    2575 scope.go:117] "RemoveContainer" containerID="2efafc58d46ca5a57f58c4f64498635765586628d1d92c5aa81b9d2ead3aa737"
Sep 15 21:44:44.601994 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:44:44.601968    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2efafc58d46ca5a57f58c4f64498635765586628d1d92c5aa81b9d2ead3aa737\": container with ID starting with 2efafc58d46ca5a57f58c4f64498635765586628d1d92c5aa81b9d2ead3aa737 not found: ID does not exist" containerID="2efafc58d46ca5a57f58c4f64498635765586628d1d92c5aa81b9d2ead3aa737"
Sep 15 21:44:44.602093 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.602009    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2efafc58d46ca5a57f58c4f64498635765586628d1d92c5aa81b9d2ead3aa737"} err="failed to get container status \"2efafc58d46ca5a57f58c4f64498635765586628d1d92c5aa81b9d2ead3aa737\": rpc error: code = NotFound desc = could not find container \"2efafc58d46ca5a57f58c4f64498635765586628d1d92c5aa81b9d2ead3aa737\": container with ID starting with 2efafc58d46ca5a57f58c4f64498635765586628d1d92c5aa81b9d2ead3aa737 not found: ID does not exist"
Sep 15 21:44:44.602093 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.602038    2575 scope.go:117] "RemoveContainer" containerID="1c68dd836cffff77a11096917b46f02819f8e37276d6a2a29450967a90bc6a3d"
Sep 15 21:44:44.611964 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.611941    2575 scope.go:117] "RemoveContainer" containerID="1c68dd836cffff77a11096917b46f02819f8e37276d6a2a29450967a90bc6a3d"
Sep 15 21:44:44.612662 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:44:44.612464    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c68dd836cffff77a11096917b46f02819f8e37276d6a2a29450967a90bc6a3d\": container with ID starting with 1c68dd836cffff77a11096917b46f02819f8e37276d6a2a29450967a90bc6a3d not found: ID does not exist" containerID="1c68dd836cffff77a11096917b46f02819f8e37276d6a2a29450967a90bc6a3d"
Sep 15 21:44:44.612662 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.612506    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c68dd836cffff77a11096917b46f02819f8e37276d6a2a29450967a90bc6a3d"} err="failed to get container status \"1c68dd836cffff77a11096917b46f02819f8e37276d6a2a29450967a90bc6a3d\": rpc error: code = NotFound desc = could not find container \"1c68dd836cffff77a11096917b46f02819f8e37276d6a2a29450967a90bc6a3d\": container with ID starting with 1c68dd836cffff77a11096917b46f02819f8e37276d6a2a29450967a90bc6a3d not found: ID does not exist"
Sep 15 21:44:44.615186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.615158    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"]
Sep 15 21:44:44.618824 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.618785    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e2d4753a-785f7bbf74-55lcr"]
Sep 15 21:44:44.631828 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.631788    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"]
Sep 15 21:44:44.635243 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.635207    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e2d4753a-7dff4bf945-qngzq"]
Sep 15 21:44:44.663195 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.663108    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4lq5\" (UniqueName: \"kubernetes.io/projected/bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd-kube-api-access-w4lq5\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:44:44.663195 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.663140    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chl7n\" (UniqueName: \"kubernetes.io/projected/27af269b-9953-4ed3-b9e4-d35fb1840e43-kube-api-access-chl7n\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:44:44.748490 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.748448    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27af269b-9953-4ed3-b9e4-d35fb1840e43" path="/var/lib/kubelet/pods/27af269b-9953-4ed3-b9e4-d35fb1840e43/volumes"
Sep 15 21:44:44.748954 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:44.748937    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd" path="/var/lib/kubelet/pods/bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd/volumes"
Sep 15 21:44:47.934823 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:47.934788    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:44:58.787850 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:58.787793    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl" podUID="1f52a028-b94f-4429-940c-1b66cee6c421" containerName="main" containerID="cri-o://6fa55a56b3aa0ca96064867c60644c7b0d160c99c1f84f9074f8cb362d079329" gracePeriod=45
Sep 15 21:44:59.052016 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.051939    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:44:59.118501 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.118461    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f52a028-b94f-4429-940c-1b66cee6c421-tls-certs\") pod \"1f52a028-b94f-4429-940c-1b66cee6c421\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") "
Sep 15 21:44:59.118501 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.118501    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-model-cache\") pod \"1f52a028-b94f-4429-940c-1b66cee6c421\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") "
Sep 15 21:44:59.118703 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.118581    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-home\") pod \"1f52a028-b94f-4429-940c-1b66cee6c421\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") "
Sep 15 21:44:59.118703 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.118630    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-tmp-dir\") pod \"1f52a028-b94f-4429-940c-1b66cee6c421\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") "
Sep 15 21:44:59.118703 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.118665    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6pgr\" (UniqueName: \"kubernetes.io/projected/1f52a028-b94f-4429-940c-1b66cee6c421-kube-api-access-p6pgr\") pod \"1f52a028-b94f-4429-940c-1b66cee6c421\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") "
Sep 15 21:44:59.118848 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.118707    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-dshm\") pod \"1f52a028-b94f-4429-940c-1b66cee6c421\" (UID: \"1f52a028-b94f-4429-940c-1b66cee6c421\") "
Sep 15 21:44:59.118915 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.118886    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f52a028-b94f-4429-940c-1b66cee6c421" (UID: "1f52a028-b94f-4429-940c-1b66cee6c421"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:59.118970 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.118907    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-home" (OuterVolumeSpecName: "home") pod "1f52a028-b94f-4429-940c-1b66cee6c421" (UID: "1f52a028-b94f-4429-940c-1b66cee6c421"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:59.119254 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.119224    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f52a028-b94f-4429-940c-1b66cee6c421" (UID: "1f52a028-b94f-4429-940c-1b66cee6c421"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:59.119374 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.119246    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:44:59.119374 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.119287    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:44:59.121558 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.121527    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-dshm" (OuterVolumeSpecName: "dshm") pod "1f52a028-b94f-4429-940c-1b66cee6c421" (UID: "1f52a028-b94f-4429-940c-1b66cee6c421"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:44:59.121683 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.121593    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f52a028-b94f-4429-940c-1b66cee6c421-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f52a028-b94f-4429-940c-1b66cee6c421" (UID: "1f52a028-b94f-4429-940c-1b66cee6c421"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:44:59.121683 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.121608    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f52a028-b94f-4429-940c-1b66cee6c421-kube-api-access-p6pgr" (OuterVolumeSpecName: "kube-api-access-p6pgr") pod "1f52a028-b94f-4429-940c-1b66cee6c421" (UID: "1f52a028-b94f-4429-940c-1b66cee6c421"). InnerVolumeSpecName "kube-api-access-p6pgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:44:59.220504 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.220455    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6pgr\" (UniqueName: \"kubernetes.io/projected/1f52a028-b94f-4429-940c-1b66cee6c421-kube-api-access-p6pgr\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:44:59.220504 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.220492    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:44:59.220504 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.220506    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f52a028-b94f-4429-940c-1b66cee6c421-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:44:59.220504 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.220518    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f52a028-b94f-4429-940c-1b66cee6c421-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:44:59.681928 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.681887    2575 generic.go:356] "Generic (PLEG): container finished" podID="1f52a028-b94f-4429-940c-1b66cee6c421" containerID="6fa55a56b3aa0ca96064867c60644c7b0d160c99c1f84f9074f8cb362d079329" exitCode=0
Sep 15 21:44:59.681928 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.681931    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"
Sep 15 21:44:59.682146 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.681977    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl" event={"ID":"1f52a028-b94f-4429-940c-1b66cee6c421","Type":"ContainerDied","Data":"6fa55a56b3aa0ca96064867c60644c7b0d160c99c1f84f9074f8cb362d079329"}
Sep 15 21:44:59.682146 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.682024    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl" event={"ID":"1f52a028-b94f-4429-940c-1b66cee6c421","Type":"ContainerDied","Data":"1c5419b51d8d62cd6c3e546f0a9df2ad46f95d3a64bd0ecf8c57e43f5e6e45dd"}
Sep 15 21:44:59.682146 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.682042    2575 scope.go:117] "RemoveContainer" containerID="6fa55a56b3aa0ca96064867c60644c7b0d160c99c1f84f9074f8cb362d079329"
Sep 15 21:44:59.692100 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.692080    2575 scope.go:117] "RemoveContainer" containerID="6fa55a56b3aa0ca96064867c60644c7b0d160c99c1f84f9074f8cb362d079329"
Sep 15 21:44:59.692465 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:44:59.692446    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa55a56b3aa0ca96064867c60644c7b0d160c99c1f84f9074f8cb362d079329\": container with ID starting with 6fa55a56b3aa0ca96064867c60644c7b0d160c99c1f84f9074f8cb362d079329 not found: ID does not exist" containerID="6fa55a56b3aa0ca96064867c60644c7b0d160c99c1f84f9074f8cb362d079329"
Sep 15 21:44:59.692537 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.692476    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa55a56b3aa0ca96064867c60644c7b0d160c99c1f84f9074f8cb362d079329"} err="failed to get container status \"6fa55a56b3aa0ca96064867c60644c7b0d160c99c1f84f9074f8cb362d079329\": rpc error: code = NotFound desc = could not find container \"6fa55a56b3aa0ca96064867c60644c7b0d160c99c1f84f9074f8cb362d079329\": container with ID starting with 6fa55a56b3aa0ca96064867c60644c7b0d160c99c1f84f9074f8cb362d079329 not found: ID does not exist"
Sep 15 21:44:59.703916 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.703878    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"]
Sep 15 21:44:59.707398 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:44:59.707364    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-4a3a26/explicit-model-4a3a26-kserve-547bfdc59f-xs2dl"]
Sep 15 21:45:00.146784 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.146723    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2d9c4-29825145-cwl78"]
Sep 15 21:45:00.147732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147348    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73" containerName="payload-pre-processing"
Sep 15 21:45:00.147732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147369    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73" containerName="payload-pre-processing"
Sep 15 21:45:00.147732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147395    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a528d4-d94b-4911-ae0b-4dc223aab4d0" containerName="payload-processing"
Sep 15 21:45:00.147732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147402    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a528d4-d94b-4911-ae0b-4dc223aab4d0" containerName="payload-processing"
Sep 15 21:45:00.147732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147413    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27af269b-9953-4ed3-b9e4-d35fb1840e43" containerName="payload-processing"
Sep 15 21:45:00.147732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147493    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="27af269b-9953-4ed3-b9e4-d35fb1840e43" containerName="payload-processing"
Sep 15 21:45:00.147732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147521    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f52a028-b94f-4429-940c-1b66cee6c421" containerName="main"
Sep 15 21:45:00.147732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147529    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f52a028-b94f-4429-940c-1b66cee6c421" containerName="main"
Sep 15 21:45:00.147732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147563    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd" containerName="payload-pre-processing"
Sep 15 21:45:00.147732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147571    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd" containerName="payload-pre-processing"
Sep 15 21:45:00.147732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147581    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0" containerName="revoke-keys"
Sep 15 21:45:00.147732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147589    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0" containerName="revoke-keys"
Sep 15 21:45:00.147732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147712    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="39a528d4-d94b-4911-ae0b-4dc223aab4d0" containerName="payload-processing"
Sep 15 21:45:00.147732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147727    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f52a028-b94f-4429-940c-1b66cee6c421" containerName="main"
Sep 15 21:45:00.147732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147736    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfc8f9c3-88bc-4e7e-b40f-57bc86d1dfdd" containerName="payload-pre-processing"
Sep 15 21:45:00.147732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147744    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="27af269b-9953-4ed3-b9e4-d35fb1840e43" containerName="payload-processing"
Sep 15 21:45:00.148362 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147781    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce48a18d-f3a5-4df4-a5db-f2d1a5d9cb73" containerName="payload-pre-processing"
Sep 15 21:45:00.148362 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.147810    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1f2fda3-c5e9-4c29-9ee3-e1805e2a55c0" containerName="revoke-keys"
Sep 15 21:45:00.152238 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.152214    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825145-mc74c"]
Sep 15 21:45:00.152365 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.152338    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2d9c4-29825145-cwl78"
Sep 15 21:45:00.160549 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.155077    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 21:45:00.164464 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.164431    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1af1db-29825145-qs5fq"]
Sep 15 21:45:00.164621 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.164579    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-mc74c"
Sep 15 21:45:00.171197 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.171156    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2d9c4-29825145-cwl78"]
Sep 15 21:45:00.172875 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.171337    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1af1db-29825145-qs5fq"
Sep 15 21:45:00.173158 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.173135    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825145-mc74c"]
Sep 15 21:45:00.175974 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.175943    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1af1db-29825145-qs5fq"]
Sep 15 21:45:00.177219 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.177196    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2d9c4-29825145-cwl78"
Sep 15 21:45:00.187682 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.187657    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-mc74c"
Sep 15 21:45:00.229629 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.229597    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1af1db-29825145-qs5fq"
Sep 15 21:45:00.457501 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.457470    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2d9c4-29825145-cwl78"]
Sep 15 21:45:00.459641 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:45:00.459612    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29a21a3_b861_45fd_87f3_9e0651d80823.slice/crio-24d606ad71152aed2b163dbc1e81762197a3f09c5089b76c615baace75d876ad WatchSource:0}: Error finding container 24d606ad71152aed2b163dbc1e81762197a3f09c5089b76c615baace75d876ad: Status 404 returned error can't find the container with id 24d606ad71152aed2b163dbc1e81762197a3f09c5089b76c615baace75d876ad
Sep 15 21:45:00.481405 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.481375    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825145-mc74c"]
Sep 15 21:45:00.484284 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:45:00.484244    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858c5f19_5948_41b1_8607_8a07a6c1ebed.slice/crio-09bc7d5eaad201c04b8da787e3a3913242768d2b1054c226b853a368e0f14229 WatchSource:0}: Error finding container 09bc7d5eaad201c04b8da787e3a3913242768d2b1054c226b853a368e0f14229: Status 404 returned error can't find the container with id 09bc7d5eaad201c04b8da787e3a3913242768d2b1054c226b853a368e0f14229
Sep 15 21:45:00.522865 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.522831    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1af1db-29825145-qs5fq"]
Sep 15 21:45:00.524909 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:45:00.524884    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8766399_c40c_453a_94d3_137282e231ca.slice/crio-0d6b521b761a6cd220a25fcb18e1712371a3b29a0c50dc6b621f95fd7dc0e449 WatchSource:0}: Error finding container 0d6b521b761a6cd220a25fcb18e1712371a3b29a0c50dc6b621f95fd7dc0e449: Status 404 returned error can't find the container with id 0d6b521b761a6cd220a25fcb18e1712371a3b29a0c50dc6b621f95fd7dc0e449
Sep 15 21:45:00.688826 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.688788    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-mc74c" event={"ID":"858c5f19-5948-41b1-8607-8a07a6c1ebed","Type":"ContainerStarted","Data":"09bc7d5eaad201c04b8da787e3a3913242768d2b1054c226b853a368e0f14229"}
Sep 15 21:45:00.691065 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.691034    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2d9c4-29825145-cwl78" event={"ID":"e29a21a3-b861-45fd-87f3-9e0651d80823","Type":"ContainerStarted","Data":"24d606ad71152aed2b163dbc1e81762197a3f09c5089b76c615baace75d876ad"}
Sep 15 21:45:00.692411 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.692388    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1af1db-29825145-qs5fq" event={"ID":"a8766399-c40c-453a-94d3-137282e231ca","Type":"ContainerStarted","Data":"0d6b521b761a6cd220a25fcb18e1712371a3b29a0c50dc6b621f95fd7dc0e449"}
Sep 15 21:45:00.748815 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:00.748718    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f52a028-b94f-4429-940c-1b66cee6c421" path="/var/lib/kubelet/pods/1f52a028-b94f-4429-940c-1b66cee6c421/volumes"
Sep 15 21:45:01.701289 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:01.701195    2575 generic.go:356] "Generic (PLEG): container finished" podID="858c5f19-5948-41b1-8607-8a07a6c1ebed" containerID="95f2b00f4e84b42f3bb508ad8cd3ad12e6599789485d6b347998cd49537ed0a4" exitCode=0
Sep 15 21:45:01.701691 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:01.701281    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-mc74c" event={"ID":"858c5f19-5948-41b1-8607-8a07a6c1ebed","Type":"ContainerDied","Data":"95f2b00f4e84b42f3bb508ad8cd3ad12e6599789485d6b347998cd49537ed0a4"}
Sep 15 21:45:01.702998 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:01.702971    2575 generic.go:356] "Generic (PLEG): container finished" podID="e29a21a3-b861-45fd-87f3-9e0651d80823" containerID="77d777d4b09f2469538fff8acb8191afbd4921002efc64a30f7fe10641437991" exitCode=0
Sep 15 21:45:01.703118 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:01.703058    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2d9c4-29825145-cwl78" event={"ID":"e29a21a3-b861-45fd-87f3-9e0651d80823","Type":"ContainerDied","Data":"77d777d4b09f2469538fff8acb8191afbd4921002efc64a30f7fe10641437991"}
Sep 15 21:45:01.704715 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:01.704692    2575 generic.go:356] "Generic (PLEG): container finished" podID="a8766399-c40c-453a-94d3-137282e231ca" containerID="0eb05330b961f4b7e37e98a27e8f5260057df26533faf7f6a41e7509acfa233f" exitCode=0
Sep 15 21:45:01.704834 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:01.704773    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1af1db-29825145-qs5fq" event={"ID":"a8766399-c40c-453a-94d3-137282e231ca","Type":"ContainerDied","Data":"0eb05330b961f4b7e37e98a27e8f5260057df26533faf7f6a41e7509acfa233f"}
Sep 15 21:45:02.866575 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:02.866541    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2d9c4-29825145-cwl78"
Sep 15 21:45:02.924931 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:02.924904    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-mc74c"
Sep 15 21:45:02.928272 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:02.928244    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1af1db-29825145-qs5fq"
Sep 15 21:45:03.717613 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:03.717581    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-mc74c"
Sep 15 21:45:03.717613 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:03.717601    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825145-mc74c" event={"ID":"858c5f19-5948-41b1-8607-8a07a6c1ebed","Type":"ContainerDied","Data":"09bc7d5eaad201c04b8da787e3a3913242768d2b1054c226b853a368e0f14229"}
Sep 15 21:45:03.717878 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:03.717683    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09bc7d5eaad201c04b8da787e3a3913242768d2b1054c226b853a368e0f14229"
Sep 15 21:45:03.719011 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:03.718990    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2d9c4-29825145-cwl78"
Sep 15 21:45:03.719126 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:03.719030    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2d9c4-29825145-cwl78" event={"ID":"e29a21a3-b861-45fd-87f3-9e0651d80823","Type":"ContainerDied","Data":"24d606ad71152aed2b163dbc1e81762197a3f09c5089b76c615baace75d876ad"}
Sep 15 21:45:03.719126 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:03.719078    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24d606ad71152aed2b163dbc1e81762197a3f09c5089b76c615baace75d876ad"
Sep 15 21:45:03.720532 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:03.720511    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1af1db-29825145-qs5fq"
Sep 15 21:45:03.720653 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:03.720556    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1af1db-29825145-qs5fq" event={"ID":"a8766399-c40c-453a-94d3-137282e231ca","Type":"ContainerDied","Data":"0d6b521b761a6cd220a25fcb18e1712371a3b29a0c50dc6b621f95fd7dc0e449"}
Sep 15 21:45:03.720653 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:03.720598    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d6b521b761a6cd220a25fcb18e1712371a3b29a0c50dc6b621f95fd7dc0e449"
Sep 15 21:45:04.403538 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.403506    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"]
Sep 15 21:45:04.404008 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.403991    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="858c5f19-5948-41b1-8607-8a07a6c1ebed" containerName="cleanup"
Sep 15 21:45:04.404070 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.404010    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="858c5f19-5948-41b1-8607-8a07a6c1ebed" containerName="cleanup"
Sep 15 21:45:04.404070 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.404023    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8766399-c40c-453a-94d3-137282e231ca" containerName="cleanup"
Sep 15 21:45:04.404070 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.404028    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8766399-c40c-453a-94d3-137282e231ca" containerName="cleanup"
Sep 15 21:45:04.404070 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.404041    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e29a21a3-b861-45fd-87f3-9e0651d80823" containerName="cleanup"
Sep 15 21:45:04.404070 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.404047    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e29a21a3-b861-45fd-87f3-9e0651d80823" containerName="cleanup"
Sep 15 21:45:04.404282 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.404120    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8766399-c40c-453a-94d3-137282e231ca" containerName="cleanup"
Sep 15 21:45:04.404282 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.404131    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e29a21a3-b861-45fd-87f3-9e0651d80823" containerName="cleanup"
Sep 15 21:45:04.404282 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.404141    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="858c5f19-5948-41b1-8607-8a07a6c1ebed" containerName="cleanup"
Sep 15 21:45:04.408733 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.408707    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.411003 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.410979    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-850f22-openshift-default-dockercfg-q989j\""
Sep 15 21:45:04.422771 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.422726    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"]
Sep 15 21:45:04.478693 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.478659    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-podinfo\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.478905 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.478700    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-token\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.478905 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.478731    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istiod-ca-cert\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.478905 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.478802    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-envoy\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.478905 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.478822    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-data\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.478905 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.478843    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-workload-certs\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.478905 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.478885    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-credential-socket\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.479129 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.478911    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vft7\" (UniqueName: \"kubernetes.io/projected/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-kube-api-access-9vft7\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.479129 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.479001    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-workload-socket\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.580320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.580280    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-podinfo\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.580320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.580320    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-token\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.580583 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.580346    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istiod-ca-cert\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.580583 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.580368    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-envoy\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.580583 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.580388    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-data\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.580583 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.580415    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-workload-certs\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.580583 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.580447    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-credential-socket\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.580583 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.580483    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vft7\" (UniqueName: \"kubernetes.io/projected/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-kube-api-access-9vft7\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.580583 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.580557    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-workload-socket\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.580957 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.580828    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-data\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.580957 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.580920    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-workload-certs\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.581036 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.580968    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-credential-socket\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.581085 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.581058    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-workload-socket\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.581216 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.581184    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istiod-ca-cert\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.583000 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.582983    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-envoy\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.583124 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.583106    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-podinfo\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.589133 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.589108    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-token\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.589272 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.589157    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vft7\" (UniqueName: \"kubernetes.io/projected/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-kube-api-access-9vft7\") pod \"e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") " pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:04.721495 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:04.721392    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:05.000606 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:05.000573    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"]
Sep 15 21:45:05.002106 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:45:05.002078    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48b61d5_9f9c_4527_b08c_c3fd2523bc98.slice/crio-c703cd519e92ad902183b857cd577c218a7c84d6195d400fdf357bc6335b1af1 WatchSource:0}: Error finding container c703cd519e92ad902183b857cd577c218a7c84d6195d400fdf357bc6335b1af1: Status 404 returned error can't find the container with id c703cd519e92ad902183b857cd577c218a7c84d6195d400fdf357bc6335b1af1
Sep 15 21:45:05.004655 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:05.004619    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:45:05.004748 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:05.004691    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:45:05.004748 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:05.004720    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:45:05.736824 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:05.736750    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q" event={"ID":"f48b61d5-9f9c-4527-b08c-c3fd2523bc98","Type":"ContainerStarted","Data":"631bc24a6282ac61000fabbaa87b0e99fe9e10f69154cb35a2da53ee1d10c320"}
Sep 15 21:45:05.737239 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:05.736833    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q" event={"ID":"f48b61d5-9f9c-4527-b08c-c3fd2523bc98","Type":"ContainerStarted","Data":"c703cd519e92ad902183b857cd577c218a7c84d6195d400fdf357bc6335b1af1"}
Sep 15 21:45:05.756810 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:05.756730    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q" podStartSLOduration=1.756706039 podStartE2EDuration="1.756706039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:45:05.756093823 +0000 UTC m=+1143.568102285" watchObservedRunningTime="2026-09-15 21:45:05.756706039 +0000 UTC m=+1143.568714485"
Sep 15 21:45:06.721628 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:06.721591    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:06.727422 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:06.727390    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:06.749243 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:06.749215    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:06.749766 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:06.749286    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:11.534234 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.534188    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"]
Sep 15 21:45:11.538415 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.538387    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"
Sep 15 21:45:11.540842 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.540813    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-850f22\""
Sep 15 21:45:11.541011 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.540867    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-850f22-dockercfg-8z464\""
Sep 15 21:45:11.548254 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.548221    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"]
Sep 15 21:45:11.566977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.566937    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"]
Sep 15 21:45:11.570625 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.570604    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"
Sep 15 21:45:11.584002 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.583968    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"]
Sep 15 21:45:11.657435 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.657395    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5sm4\" (UniqueName: \"kubernetes.io/projected/f0ad6e4f-bfad-48fb-943c-183c31673c1b-kube-api-access-r5sm4\") pod \"payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg\" (UID: \"f0ad6e4f-bfad-48fb-943c-183c31673c1b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"
Sep 15 21:45:11.657657 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.657562    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ad6e4f-bfad-48fb-943c-183c31673c1b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg\" (UID: \"f0ad6e4f-bfad-48fb-943c-183c31673c1b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"
Sep 15 21:45:11.657723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.657698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztskb\" (UniqueName: \"kubernetes.io/projected/87fd9284-2fa6-4d55-83e1-6fe45b4957c5-kube-api-access-ztskb\") pod \"payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb\" (UID: \"87fd9284-2fa6-4d55-83e1-6fe45b4957c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"
Sep 15 21:45:11.657831 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.657737    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87fd9284-2fa6-4d55-83e1-6fe45b4957c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb\" (UID: \"87fd9284-2fa6-4d55-83e1-6fe45b4957c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"
Sep 15 21:45:11.758632 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.758573    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ad6e4f-bfad-48fb-943c-183c31673c1b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg\" (UID: \"f0ad6e4f-bfad-48fb-943c-183c31673c1b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"
Sep 15 21:45:11.758854 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.758679    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztskb\" (UniqueName: \"kubernetes.io/projected/87fd9284-2fa6-4d55-83e1-6fe45b4957c5-kube-api-access-ztskb\") pod \"payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb\" (UID: \"87fd9284-2fa6-4d55-83e1-6fe45b4957c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"
Sep 15 21:45:11.758854 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.758700    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87fd9284-2fa6-4d55-83e1-6fe45b4957c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb\" (UID: \"87fd9284-2fa6-4d55-83e1-6fe45b4957c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"
Sep 15 21:45:11.758854 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.758747    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5sm4\" (UniqueName: \"kubernetes.io/projected/f0ad6e4f-bfad-48fb-943c-183c31673c1b-kube-api-access-r5sm4\") pod \"payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg\" (UID: \"f0ad6e4f-bfad-48fb-943c-183c31673c1b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"
Sep 15 21:45:11.759567 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.759543    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87fd9284-2fa6-4d55-83e1-6fe45b4957c5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb\" (UID: \"87fd9284-2fa6-4d55-83e1-6fe45b4957c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"
Sep 15 21:45:11.759611 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.759543    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ad6e4f-bfad-48fb-943c-183c31673c1b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg\" (UID: \"f0ad6e4f-bfad-48fb-943c-183c31673c1b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"
Sep 15 21:45:11.766736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.766701    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5sm4\" (UniqueName: \"kubernetes.io/projected/f0ad6e4f-bfad-48fb-943c-183c31673c1b-kube-api-access-r5sm4\") pod \"payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg\" (UID: \"f0ad6e4f-bfad-48fb-943c-183c31673c1b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"
Sep 15 21:45:11.766912 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.766810    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztskb\" (UniqueName: \"kubernetes.io/projected/87fd9284-2fa6-4d55-83e1-6fe45b4957c5-kube-api-access-ztskb\") pod \"payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb\" (UID: \"87fd9284-2fa6-4d55-83e1-6fe45b4957c5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"
Sep 15 21:45:11.851948 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.851913    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"
Sep 15 21:45:11.893395 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:11.893358    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"
Sep 15 21:45:12.129027 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:12.128998    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"]
Sep 15 21:45:12.131198 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:45:12.131172    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87fd9284_2fa6_4d55_83e1_6fe45b4957c5.slice/crio-0ec7049cc4d1e7486b0dbbf209e0fbc69f3b2116db1f7515077e027f74aa3740 WatchSource:0}: Error finding container 0ec7049cc4d1e7486b0dbbf209e0fbc69f3b2116db1f7515077e027f74aa3740: Status 404 returned error can't find the container with id 0ec7049cc4d1e7486b0dbbf209e0fbc69f3b2116db1f7515077e027f74aa3740
Sep 15 21:45:12.181629 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:12.181573    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"]
Sep 15 21:45:12.185996 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:45:12.185965    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ad6e4f_bfad_48fb_943c_183c31673c1b.slice/crio-7b7a90e6923b4f9769cb27c7eef9504e7f7b95d86746c96a0805e8df28710ac8 WatchSource:0}: Error finding container 7b7a90e6923b4f9769cb27c7eef9504e7f7b95d86746c96a0805e8df28710ac8: Status 404 returned error can't find the container with id 7b7a90e6923b4f9769cb27c7eef9504e7f7b95d86746c96a0805e8df28710ac8
Sep 15 21:45:12.780105 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:12.780054    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg" event={"ID":"f0ad6e4f-bfad-48fb-943c-183c31673c1b","Type":"ContainerStarted","Data":"e0b0be6b4f9b7c379370d67a658b2b8101106419fb5633efca038f28e25a7c78"}
Sep 15 21:45:12.780535 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:12.780128    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg" event={"ID":"f0ad6e4f-bfad-48fb-943c-183c31673c1b","Type":"ContainerStarted","Data":"7b7a90e6923b4f9769cb27c7eef9504e7f7b95d86746c96a0805e8df28710ac8"}
Sep 15 21:45:12.780535 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:12.780253    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"
Sep 15 21:45:12.782001 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:12.781968    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb" event={"ID":"87fd9284-2fa6-4d55-83e1-6fe45b4957c5","Type":"ContainerStarted","Data":"286ba67327208d1000b5426b06772d819578a57284da184113cb1feff9efcff5"}
Sep 15 21:45:12.782163 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:12.782025    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb" event={"ID":"87fd9284-2fa6-4d55-83e1-6fe45b4957c5","Type":"ContainerStarted","Data":"0ec7049cc4d1e7486b0dbbf209e0fbc69f3b2116db1f7515077e027f74aa3740"}
Sep 15 21:45:12.782163 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:12.782049    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"
Sep 15 21:45:12.795185 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:12.795133    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg" podStartSLOduration=1.795111962 podStartE2EDuration="1.795111962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:45:12.794641902 +0000 UTC m=+1150.606650350" watchObservedRunningTime="2026-09-15 21:45:12.795111962 +0000 UTC m=+1150.607120408"
Sep 15 21:45:12.810019 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:12.809964    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb" podStartSLOduration=1.8099451960000001 podStartE2EDuration="1.809945196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:45:12.80883313 +0000 UTC m=+1150.620841576" watchObservedRunningTime="2026-09-15 21:45:12.809945196 +0000 UTC m=+1150.621953640"
Sep 15 21:45:16.336397 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:16.336350    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:45:23.790903 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:23.790819    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"
Sep 15 21:45:23.790903 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:23.790891    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"
Sep 15 21:45:35.613724 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:35.613682    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"]
Sep 15 21:45:35.614162 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:35.613991    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8" podUID="ae7cd896-0afa-48d3-a34e-a74f7b3342f9" containerName="istio-proxy" containerID="cri-o://0a49629f0d1d79e0e71627d565141bb28f5764e3499d5c1912e14ef4e61f9a8b" gracePeriod=30
Sep 15 21:45:37.421930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.421882    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"]
Sep 15 21:45:37.426127 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.426105    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.428279 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.428253    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-56e305\"/\"e2e-conflict-a-d70fe03c-kserve-self-signed-certs\""
Sep 15 21:45:37.432977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.432950    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"]
Sep 15 21:45:37.533206 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.533162    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-home\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.533404 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.533267    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-tmp-dir\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.533404 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.533285    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-dshm\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.533404 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.533313    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e285a82f-a400-4857-ad57-2edec5487452-tls-certs\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.533508 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.533441    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-model-cache\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.533508 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.533484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pgk6\" (UniqueName: \"kubernetes.io/projected/e285a82f-a400-4857-ad57-2edec5487452-kube-api-access-8pgk6\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.634821 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.634779    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pgk6\" (UniqueName: \"kubernetes.io/projected/e285a82f-a400-4857-ad57-2edec5487452-kube-api-access-8pgk6\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.635003 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.634848    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-home\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.635003 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.634919    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-tmp-dir\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.635003 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.634946    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-dshm\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.635003 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.634982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e285a82f-a400-4857-ad57-2edec5487452-tls-certs\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.635143 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.635059    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-model-cache\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.635366 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.635344    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-home\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.635432 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.635393    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-model-cache\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.635542 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.635518    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-tmp-dir\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.637404 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.637385    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-dshm\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.637660 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.637641    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e285a82f-a400-4857-ad57-2edec5487452-tls-certs\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.642309 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.642283    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pgk6\" (UniqueName: \"kubernetes.io/projected/e285a82f-a400-4857-ad57-2edec5487452-kube-api-access-8pgk6\") pod \"e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:37.738011 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:37.737913    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:38.071129 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:38.071039    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"]
Sep 15 21:45:38.078103 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:45:38.077257    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode285a82f_a400_4857_ad57_2edec5487452.slice/crio-f8e80b571f1eb5812a7c32b9e61c6385a2b3ad0106c1f660172b95a053d26872 WatchSource:0}: Error finding container f8e80b571f1eb5812a7c32b9e61c6385a2b3ad0106c1f660172b95a053d26872: Status 404 returned error can't find the container with id f8e80b571f1eb5812a7c32b9e61c6385a2b3ad0106c1f660172b95a053d26872
Sep 15 21:45:38.945174 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:38.945140    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k" event={"ID":"e285a82f-a400-4857-ad57-2edec5487452","Type":"ContainerStarted","Data":"b47001eb1918d084b2e50830050879fc154f74e9ea5b0ff26116f234cb443c35"}
Sep 15 21:45:38.945174 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:38.945184    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k" event={"ID":"e285a82f-a400-4857-ad57-2edec5487452","Type":"ContainerStarted","Data":"f8e80b571f1eb5812a7c32b9e61c6385a2b3ad0106c1f660172b95a053d26872"}
Sep 15 21:45:38.967133 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:38.967060    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k" podStartSLOduration=1.967033744 podStartE2EDuration="1.967033744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:45:38.962789545 +0000 UTC m=+1176.774797990" watchObservedRunningTime="2026-09-15 21:45:38.967033744 +0000 UTC m=+1176.779042192"
Sep 15 21:45:40.881386 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.881311    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:45:40.958867 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.958830    2575 generic.go:356] "Generic (PLEG): container finished" podID="ae7cd896-0afa-48d3-a34e-a74f7b3342f9" containerID="0a49629f0d1d79e0e71627d565141bb28f5764e3499d5c1912e14ef4e61f9a8b" exitCode=0
Sep 15 21:45:40.959045 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.958910    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"
Sep 15 21:45:40.959045 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.958910    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8" event={"ID":"ae7cd896-0afa-48d3-a34e-a74f7b3342f9","Type":"ContainerDied","Data":"0a49629f0d1d79e0e71627d565141bb28f5764e3499d5c1912e14ef4e61f9a8b"}
Sep 15 21:45:40.959176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.959052    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8" event={"ID":"ae7cd896-0afa-48d3-a34e-a74f7b3342f9","Type":"ContainerDied","Data":"f64b15254af6256ce0bfe25300a794f3b55cf67140be7eb5eebd427136c94f00"}
Sep 15 21:45:40.959176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.959076    2575 scope.go:117] "RemoveContainer" containerID="0a49629f0d1d79e0e71627d565141bb28f5764e3499d5c1912e14ef4e61f9a8b"
Sep 15 21:45:40.971688 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.971665    2575 scope.go:117] "RemoveContainer" containerID="eaf16962641130eed0135d23ea5cd40123c1cb5bbb0c8423017a5402cb8a3f23"
Sep 15 21:45:40.977312 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.977282    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-token\") pod \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") "
Sep 15 21:45:40.977476 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.977335    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp896\" (UniqueName: \"kubernetes.io/projected/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-kube-api-access-mp896\") pod \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") "
Sep 15 21:45:40.977476 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.977387    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-workload-certs\") pod \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") "
Sep 15 21:45:40.977476 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.977461    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-workload-socket\") pod \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") "
Sep 15 21:45:40.977615 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.977517    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-envoy\") pod \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") "
Sep 15 21:45:40.977615 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.977551    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istiod-ca-cert\") pod \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") "
Sep 15 21:45:40.977615 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.977581    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-credential-socket\") pod \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") "
Sep 15 21:45:40.977782 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.977632    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-podinfo\") pod \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") "
Sep 15 21:45:40.977782 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.977657    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-data\") pod \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\" (UID: \"ae7cd896-0afa-48d3-a34e-a74f7b3342f9\") "
Sep 15 21:45:40.977782 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.977700    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ae7cd896-0afa-48d3-a34e-a74f7b3342f9" (UID: "ae7cd896-0afa-48d3-a34e-a74f7b3342f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:40.978133 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.978108    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:45:40.978307 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.978260    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ae7cd896-0afa-48d3-a34e-a74f7b3342f9" (UID: "ae7cd896-0afa-48d3-a34e-a74f7b3342f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:40.978406 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.978325    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ae7cd896-0afa-48d3-a34e-a74f7b3342f9" (UID: "ae7cd896-0afa-48d3-a34e-a74f7b3342f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:45:40.978484 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.978335    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-data" (OuterVolumeSpecName: "istio-data") pod "ae7cd896-0afa-48d3-a34e-a74f7b3342f9" (UID: "ae7cd896-0afa-48d3-a34e-a74f7b3342f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:40.978563 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.978544    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ae7cd896-0afa-48d3-a34e-a74f7b3342f9" (UID: "ae7cd896-0afa-48d3-a34e-a74f7b3342f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:40.980227 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.980194    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-token" (OuterVolumeSpecName: "istio-token") pod "ae7cd896-0afa-48d3-a34e-a74f7b3342f9" (UID: "ae7cd896-0afa-48d3-a34e-a74f7b3342f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:45:40.980355 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.980247    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-kube-api-access-mp896" (OuterVolumeSpecName: "kube-api-access-mp896") pod "ae7cd896-0afa-48d3-a34e-a74f7b3342f9" (UID: "ae7cd896-0afa-48d3-a34e-a74f7b3342f9"). InnerVolumeSpecName "kube-api-access-mp896". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:45:40.980844 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.980815    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ae7cd896-0afa-48d3-a34e-a74f7b3342f9" (UID: "ae7cd896-0afa-48d3-a34e-a74f7b3342f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:45:40.980959 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.980937    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ae7cd896-0afa-48d3-a34e-a74f7b3342f9" (UID: "ae7cd896-0afa-48d3-a34e-a74f7b3342f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:45:40.983007 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.982985    2575 scope.go:117] "RemoveContainer" containerID="0a49629f0d1d79e0e71627d565141bb28f5764e3499d5c1912e14ef4e61f9a8b"
Sep 15 21:45:40.983407 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:45:40.983387    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a49629f0d1d79e0e71627d565141bb28f5764e3499d5c1912e14ef4e61f9a8b\": container with ID starting with 0a49629f0d1d79e0e71627d565141bb28f5764e3499d5c1912e14ef4e61f9a8b not found: ID does not exist" containerID="0a49629f0d1d79e0e71627d565141bb28f5764e3499d5c1912e14ef4e61f9a8b"
Sep 15 21:45:40.983506 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.983416    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a49629f0d1d79e0e71627d565141bb28f5764e3499d5c1912e14ef4e61f9a8b"} err="failed to get container status \"0a49629f0d1d79e0e71627d565141bb28f5764e3499d5c1912e14ef4e61f9a8b\": rpc error: code = NotFound desc = could not find container \"0a49629f0d1d79e0e71627d565141bb28f5764e3499d5c1912e14ef4e61f9a8b\": container with ID starting with 0a49629f0d1d79e0e71627d565141bb28f5764e3499d5c1912e14ef4e61f9a8b not found: ID does not exist"
Sep 15 21:45:40.983506 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.983435    2575 scope.go:117] "RemoveContainer" containerID="eaf16962641130eed0135d23ea5cd40123c1cb5bbb0c8423017a5402cb8a3f23"
Sep 15 21:45:40.983824 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:45:40.983737    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf16962641130eed0135d23ea5cd40123c1cb5bbb0c8423017a5402cb8a3f23\": container with ID starting with eaf16962641130eed0135d23ea5cd40123c1cb5bbb0c8423017a5402cb8a3f23 not found: ID does not exist" containerID="eaf16962641130eed0135d23ea5cd40123c1cb5bbb0c8423017a5402cb8a3f23"
Sep 15 21:45:40.983824 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:40.983790    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf16962641130eed0135d23ea5cd40123c1cb5bbb0c8423017a5402cb8a3f23"} err="failed to get container status \"eaf16962641130eed0135d23ea5cd40123c1cb5bbb0c8423017a5402cb8a3f23\": rpc error: code = NotFound desc = could not find container \"eaf16962641130eed0135d23ea5cd40123c1cb5bbb0c8423017a5402cb8a3f23\": container with ID starting with eaf16962641130eed0135d23ea5cd40123c1cb5bbb0c8423017a5402cb8a3f23 not found: ID does not exist"
Sep 15 21:45:41.079662 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:41.079628    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:45:41.079662 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:41.079659    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:45:41.079662 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:41.079669    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:45:41.079931 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:41.079678    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:45:41.079931 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:41.079686    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:45:41.079931 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:41.079694    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp896\" (UniqueName: \"kubernetes.io/projected/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-kube-api-access-mp896\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:45:41.079931 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:41.079704    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:45:41.079931 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:41.079712    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae7cd896-0afa-48d3-a34e-a74f7b3342f9-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:45:41.283542 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:41.283505    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"]
Sep 15 21:45:41.287475 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:41.287441    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-4a3a26-openshift-default-65f5b98664-b7sn8"]
Sep 15 21:45:42.748115 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:42.748078    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae7cd896-0afa-48d3-a34e-a74f7b3342f9" path="/var/lib/kubelet/pods/ae7cd896-0afa-48d3-a34e-a74f7b3342f9/volumes"
Sep 15 21:45:43.981401 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:43.981311    2575 generic.go:356] "Generic (PLEG): container finished" podID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" containerID="631bc24a6282ac61000fabbaa87b0e99fe9e10f69154cb35a2da53ee1d10c320" exitCode=0
Sep 15 21:45:43.981401 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:43.981381    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q" event={"ID":"f48b61d5-9f9c-4527-b08c-c3fd2523bc98","Type":"ContainerDied","Data":"631bc24a6282ac61000fabbaa87b0e99fe9e10f69154cb35a2da53ee1d10c320"}
Sep 15 21:45:43.981908 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:43.981893    2575 scope.go:117] "RemoveContainer" containerID="631bc24a6282ac61000fabbaa87b0e99fe9e10f69154cb35a2da53ee1d10c320"
Sep 15 21:45:43.983185 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:43.983160    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:45:43.983255 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:43.983228    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:45:43.983292 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:43.983259    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:45:44.989152 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:44.989113    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q" event={"ID":"f48b61d5-9f9c-4527-b08c-c3fd2523bc98","Type":"ContainerStarted","Data":"9932d3be156c2352ed062afd480d660db95c49bece7e2cc20ae7ae4a286c1838"}
Sep 15 21:45:45.722543 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.722494    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:45.722790 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.722744    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 15 21:45:45.722858 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.722827    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q" podUID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 15 21:45:45.862223 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.862185    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"]
Sep 15 21:45:45.862654 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.862640    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae7cd896-0afa-48d3-a34e-a74f7b3342f9" containerName="istio-proxy"
Sep 15 21:45:45.862708 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.862655    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae7cd896-0afa-48d3-a34e-a74f7b3342f9" containerName="istio-proxy"
Sep 15 21:45:45.862708 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.862674    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae7cd896-0afa-48d3-a34e-a74f7b3342f9" containerName="istio-proxy"
Sep 15 21:45:45.862708 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.862680    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae7cd896-0afa-48d3-a34e-a74f7b3342f9" containerName="istio-proxy"
Sep 15 21:45:45.862840 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.862778    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae7cd896-0afa-48d3-a34e-a74f7b3342f9" containerName="istio-proxy"
Sep 15 21:45:45.864965 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.864948    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"
Sep 15 21:45:45.867064 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.867040    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-8c1d2a\""
Sep 15 21:45:45.867209 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.867040    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-8c1d2a-dockercfg-pdjz7\""
Sep 15 21:45:45.878362 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.878328    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"]
Sep 15 21:45:45.895780 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.895717    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"]
Sep 15 21:45:45.903156 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.903118    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae7cd896-0afa-48d3-a34e-a74f7b3342f9" containerName="istio-proxy"
Sep 15 21:45:45.919559 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.919508    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"]
Sep 15 21:45:45.919798 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:45.919668    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"
Sep 15 21:45:46.037153 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.037058    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk9cz\" (UniqueName: \"kubernetes.io/projected/2d75e36f-b1a1-4131-a27d-60989af5d770-kube-api-access-vk9cz\") pod \"payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb\" (UID: \"2d75e36f-b1a1-4131-a27d-60989af5d770\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"
Sep 15 21:45:46.037153 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.037114    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f9abe4b-1c48-421c-aed8-623679c0f32f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx\" (UID: \"2f9abe4b-1c48-421c-aed8-623679c0f32f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"
Sep 15 21:45:46.037153 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.037139    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d75e36f-b1a1-4131-a27d-60989af5d770-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb\" (UID: \"2d75e36f-b1a1-4131-a27d-60989af5d770\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"
Sep 15 21:45:46.037639 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.037158    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ltwn\" (UniqueName: \"kubernetes.io/projected/2f9abe4b-1c48-421c-aed8-623679c0f32f-kube-api-access-7ltwn\") pod \"payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx\" (UID: \"2f9abe4b-1c48-421c-aed8-623679c0f32f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"
Sep 15 21:45:46.138578 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.138539    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk9cz\" (UniqueName: \"kubernetes.io/projected/2d75e36f-b1a1-4131-a27d-60989af5d770-kube-api-access-vk9cz\") pod \"payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb\" (UID: \"2d75e36f-b1a1-4131-a27d-60989af5d770\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"
Sep 15 21:45:46.138815 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.138606    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f9abe4b-1c48-421c-aed8-623679c0f32f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx\" (UID: \"2f9abe4b-1c48-421c-aed8-623679c0f32f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"
Sep 15 21:45:46.138815 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.138639    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d75e36f-b1a1-4131-a27d-60989af5d770-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb\" (UID: \"2d75e36f-b1a1-4131-a27d-60989af5d770\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"
Sep 15 21:45:46.138815 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.138673    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ltwn\" (UniqueName: \"kubernetes.io/projected/2f9abe4b-1c48-421c-aed8-623679c0f32f-kube-api-access-7ltwn\") pod \"payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx\" (UID: \"2f9abe4b-1c48-421c-aed8-623679c0f32f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"
Sep 15 21:45:46.139368 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.139347    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d75e36f-b1a1-4131-a27d-60989af5d770-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb\" (UID: \"2d75e36f-b1a1-4131-a27d-60989af5d770\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"
Sep 15 21:45:46.139633 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.139614    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f9abe4b-1c48-421c-aed8-623679c0f32f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx\" (UID: \"2f9abe4b-1c48-421c-aed8-623679c0f32f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"
Sep 15 21:45:46.146014 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.145985    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk9cz\" (UniqueName: \"kubernetes.io/projected/2d75e36f-b1a1-4131-a27d-60989af5d770-kube-api-access-vk9cz\") pod \"payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb\" (UID: \"2d75e36f-b1a1-4131-a27d-60989af5d770\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"
Sep 15 21:45:46.146156 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.146109    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ltwn\" (UniqueName: \"kubernetes.io/projected/2f9abe4b-1c48-421c-aed8-623679c0f32f-kube-api-access-7ltwn\") pod \"payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx\" (UID: \"2f9abe4b-1c48-421c-aed8-623679c0f32f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"
Sep 15 21:45:46.183414 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.183366    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"
Sep 15 21:45:46.235942 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.235897    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"
Sep 15 21:45:46.475804 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.468090    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"]
Sep 15 21:45:46.480906 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:45:46.480865    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9abe4b_1c48_421c_aed8_623679c0f32f.slice/crio-b01fbca0926ca5ccb5d0c2fefb166822c578dd2d5fda8544f105278d894276ed WatchSource:0}: Error finding container b01fbca0926ca5ccb5d0c2fefb166822c578dd2d5fda8544f105278d894276ed: Status 404 returned error can't find the container with id b01fbca0926ca5ccb5d0c2fefb166822c578dd2d5fda8544f105278d894276ed
Sep 15 21:45:46.567125 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.564026    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"]
Sep 15 21:45:46.722413 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.722359    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 15 21:45:46.722640 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:46.722430    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q" podUID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 15 21:45:47.005069 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:47.005030    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb" event={"ID":"2d75e36f-b1a1-4131-a27d-60989af5d770","Type":"ContainerStarted","Data":"c7a6aed7144c17a81c643a045ef11346ddc2cc09f936015d81bc0ccfd91f83dc"}
Sep 15 21:45:47.005272 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:47.005094    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb" event={"ID":"2d75e36f-b1a1-4131-a27d-60989af5d770","Type":"ContainerStarted","Data":"2ce239c20f06ef803bde4d40f397eeabfafc11a67c2852fc0dafda0054862ceb"}
Sep 15 21:45:47.005272 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:47.005205    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"
Sep 15 21:45:47.007088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:47.007061    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx" event={"ID":"2f9abe4b-1c48-421c-aed8-623679c0f32f","Type":"ContainerStarted","Data":"d93363455a48f98900d2600afd05f1a57481756f43ca9cf42094db304fd25da5"}
Sep 15 21:45:47.007212 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:47.007105    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx" event={"ID":"2f9abe4b-1c48-421c-aed8-623679c0f32f","Type":"ContainerStarted","Data":"b01fbca0926ca5ccb5d0c2fefb166822c578dd2d5fda8544f105278d894276ed"}
Sep 15 21:45:47.007212 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:47.007124    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"
Sep 15 21:45:47.023216 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:47.023152    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb" podStartSLOduration=2.023132243 podStartE2EDuration="2.023132243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:45:47.020698702 +0000 UTC m=+1184.832707147" watchObservedRunningTime="2026-09-15 21:45:47.023132243 +0000 UTC m=+1184.835140686"
Sep 15 21:45:47.042482 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:47.042414    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx" podStartSLOduration=2.042395033 podStartE2EDuration="2.042395033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:45:47.039270097 +0000 UTC m=+1184.851278542" watchObservedRunningTime="2026-09-15 21:45:47.042395033 +0000 UTC m=+1184.854403477"
Sep 15 21:45:47.722489 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:47.722452    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 15 21:45:47.722688 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:47.722515    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q" podUID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 15 21:45:47.738920 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:47.738890    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:47.739036 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:47.738945    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:47.765870 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:47.765830    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:48.025212 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:48.025120    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:45:48.036656 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:48.036616    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:45:48.722134 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:48.722082    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 15 21:45:48.722526 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:48.722149    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q" podUID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 15 21:45:49.722815 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:49.722736    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 15 21:45:49.723216 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:49.722839    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q" podUID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 15 21:45:50.722876 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:50.722831    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Sep 15 21:45:50.723284 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:50.722909    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q" podUID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Sep 15 21:45:51.727050 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:51.727018    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:51.727464 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:51.727313    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:51.728055 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:51.728038    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:45:58.014662 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:58.014628    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"
Sep 15 21:45:58.015164 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:45:58.015017    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"
Sep 15 21:46:03.537934 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:03.537902    2575 scope.go:117] "RemoveContainer" containerID="87c39467f9ab4d4e98447b8cdd25e62cdd0f3e47791cec2b0c2b4425ca887fb2"
Sep 15 21:46:09.829821 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:09.829778    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:46:12.422700 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:12.422659    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7725-c9bb9c"]
Sep 15 21:46:12.427887 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:12.427864    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-c9bb9c"
Sep 15 21:46:12.430467 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:12.430438    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2lhf2\""
Sep 15 21:46:12.432647 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:12.432620    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7725-c9bb9c"]
Sep 15 21:46:12.529004 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:12.528964    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2z8w\" (UniqueName: \"kubernetes.io/projected/5a459ad0-26bd-4271-ba2b-ddd586d1e6a9-kube-api-access-w2z8w\") pod \"test-curl-7725-c9bb9c\" (UID: \"5a459ad0-26bd-4271-ba2b-ddd586d1e6a9\") " pod="opendatahub/test-curl-7725-c9bb9c"
Sep 15 21:46:12.630052 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:12.630014    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2z8w\" (UniqueName: \"kubernetes.io/projected/5a459ad0-26bd-4271-ba2b-ddd586d1e6a9-kube-api-access-w2z8w\") pod \"test-curl-7725-c9bb9c\" (UID: \"5a459ad0-26bd-4271-ba2b-ddd586d1e6a9\") " pod="opendatahub/test-curl-7725-c9bb9c"
Sep 15 21:46:12.638203 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:12.638162    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2z8w\" (UniqueName: \"kubernetes.io/projected/5a459ad0-26bd-4271-ba2b-ddd586d1e6a9-kube-api-access-w2z8w\") pod \"test-curl-7725-c9bb9c\" (UID: \"5a459ad0-26bd-4271-ba2b-ddd586d1e6a9\") " pod="opendatahub/test-curl-7725-c9bb9c"
Sep 15 21:46:12.739448 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:12.739348    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-c9bb9c"
Sep 15 21:46:13.015589 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:13.015566    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7725-c9bb9c"]
Sep 15 21:46:13.018476 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:46:13.018448    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a459ad0_26bd_4271_ba2b_ddd586d1e6a9.slice/crio-7e673e008e7b9f6a501af28493b1e50eae578968a1a3b562e505e64114849516 WatchSource:0}: Error finding container 7e673e008e7b9f6a501af28493b1e50eae578968a1a3b562e505e64114849516: Status 404 returned error can't find the container with id 7e673e008e7b9f6a501af28493b1e50eae578968a1a3b562e505e64114849516
Sep 15 21:46:13.175846 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:13.175806    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7725-c9bb9c" event={"ID":"5a459ad0-26bd-4271-ba2b-ddd586d1e6a9","Type":"ContainerStarted","Data":"32e698ec2def8c7181b38a02d6cfa2fbb8114662c8ec5c4813cd402b2a42ff34"}
Sep 15 21:46:13.175846 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:13.175854    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7725-c9bb9c" event={"ID":"5a459ad0-26bd-4271-ba2b-ddd586d1e6a9","Type":"ContainerStarted","Data":"7e673e008e7b9f6a501af28493b1e50eae578968a1a3b562e505e64114849516"}
Sep 15 21:46:13.190369 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:13.190316    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7725-c9bb9c" podStartSLOduration=1.190297242 podStartE2EDuration="1.190297242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:46:13.189268664 +0000 UTC m=+1211.001277107" watchObservedRunningTime="2026-09-15 21:46:13.190297242 +0000 UTC m=+1211.002305687"
Sep 15 21:46:13.815432 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:13.815341    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7725-c9bb9c"]
Sep 15 21:46:13.819324 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:13.819292    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7725-c9bb9c"]
Sep 15 21:46:13.933544 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:13.933509    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7725-c9e4c7"]
Sep 15 21:46:13.934048 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:13.934033    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a459ad0-26bd-4271-ba2b-ddd586d1e6a9" containerName="test-curl-7725-c9bb9c"
Sep 15 21:46:13.934107 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:13.934050    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a459ad0-26bd-4271-ba2b-ddd586d1e6a9" containerName="test-curl-7725-c9bb9c"
Sep 15 21:46:13.934167 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:13.934158    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a459ad0-26bd-4271-ba2b-ddd586d1e6a9" containerName="test-curl-7725-c9bb9c"
Sep 15 21:46:13.937505 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:13.937461    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-c9e4c7"
Sep 15 21:46:13.941088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:13.941058    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7725-c9e4c7"]
Sep 15 21:46:14.045346 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:14.045306    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrjql\" (UniqueName: \"kubernetes.io/projected/1b26f70c-1b8e-4a9d-ab57-c6d515ec320d-kube-api-access-nrjql\") pod \"test-curl-7725-c9e4c7\" (UID: \"1b26f70c-1b8e-4a9d-ab57-c6d515ec320d\") " pod="opendatahub/test-curl-7725-c9e4c7"
Sep 15 21:46:14.146739 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:14.146701    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjql\" (UniqueName: \"kubernetes.io/projected/1b26f70c-1b8e-4a9d-ab57-c6d515ec320d-kube-api-access-nrjql\") pod \"test-curl-7725-c9e4c7\" (UID: \"1b26f70c-1b8e-4a9d-ab57-c6d515ec320d\") " pod="opendatahub/test-curl-7725-c9e4c7"
Sep 15 21:46:14.155116 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:14.155084    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrjql\" (UniqueName: \"kubernetes.io/projected/1b26f70c-1b8e-4a9d-ab57-c6d515ec320d-kube-api-access-nrjql\") pod \"test-curl-7725-c9e4c7\" (UID: \"1b26f70c-1b8e-4a9d-ab57-c6d515ec320d\") " pod="opendatahub/test-curl-7725-c9e4c7"
Sep 15 21:46:14.182383 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:46:14.182347    2575 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7725-c9bb9c" container="test-curl-7725-c9bb9c"
Sep 15 21:46:14.250341 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:14.250301    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-c9e4c7"
Sep 15 21:46:14.528732 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:46:14.527474    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b26f70c_1b8e_4a9d_ab57_c6d515ec320d.slice/crio-60cf8c4d760b31b3002cd158c0c6d992d9892fcfca536167d8b8ae7f1d89e17c WatchSource:0}: Error finding container 60cf8c4d760b31b3002cd158c0c6d992d9892fcfca536167d8b8ae7f1d89e17c: Status 404 returned error can't find the container with id 60cf8c4d760b31b3002cd158c0c6d992d9892fcfca536167d8b8ae7f1d89e17c
Sep 15 21:46:14.529784 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:14.529707    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7725-c9e4c7"]
Sep 15 21:46:15.190365 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:15.190311    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7725-c9e4c7" event={"ID":"1b26f70c-1b8e-4a9d-ab57-c6d515ec320d","Type":"ContainerStarted","Data":"1ac6e3b1317dac89b0eab6481778b9b8cfe2c8f8cd52afb0d444953089cd36f0"}
Sep 15 21:46:15.190846 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:15.190372    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7725-c9e4c7" event={"ID":"1b26f70c-1b8e-4a9d-ab57-c6d515ec320d","Type":"ContainerStarted","Data":"60cf8c4d760b31b3002cd158c0c6d992d9892fcfca536167d8b8ae7f1d89e17c"}
Sep 15 21:46:15.190846 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:15.190485    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7725-c9bb9c" podUID="5a459ad0-26bd-4271-ba2b-ddd586d1e6a9" containerName="test-curl-7725-c9bb9c" containerID="cri-o://32e698ec2def8c7181b38a02d6cfa2fbb8114662c8ec5c4813cd402b2a42ff34" gracePeriod=2
Sep 15 21:46:15.205554 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:15.205497    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7725-c9e4c7" podStartSLOduration=2.205477048 podStartE2EDuration="2.205477048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:46:15.20294214 +0000 UTC m=+1213.014950585" watchObservedRunningTime="2026-09-15 21:46:15.205477048 +0000 UTC m=+1213.017485493"
Sep 15 21:46:15.812097 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:15.812057    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7725-c9e4c7"]
Sep 15 21:46:15.816070 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:15.816031    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7725-c9e4c7"]
Sep 15 21:46:15.935765 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:15.935701    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7725-8a419a"]
Sep 15 21:46:15.936314 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:15.936292    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b26f70c-1b8e-4a9d-ab57-c6d515ec320d" containerName="test-curl-7725-c9e4c7"
Sep 15 21:46:15.936419 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:15.936317    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b26f70c-1b8e-4a9d-ab57-c6d515ec320d" containerName="test-curl-7725-c9e4c7"
Sep 15 21:46:15.936478 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:15.936470    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b26f70c-1b8e-4a9d-ab57-c6d515ec320d" containerName="test-curl-7725-c9e4c7"
Sep 15 21:46:15.939939 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:15.939898    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-8a419a"
Sep 15 21:46:15.943232 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:15.943203    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7725-8a419a"]
Sep 15 21:46:16.066054 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:16.066008    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgv6n\" (UniqueName: \"kubernetes.io/projected/06c18733-eea4-42ce-96a0-0d3816ef029e-kube-api-access-wgv6n\") pod \"test-curl-7725-8a419a\" (UID: \"06c18733-eea4-42ce-96a0-0d3816ef029e\") " pod="opendatahub/test-curl-7725-8a419a"
Sep 15 21:46:16.167144 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:16.167096    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgv6n\" (UniqueName: \"kubernetes.io/projected/06c18733-eea4-42ce-96a0-0d3816ef029e-kube-api-access-wgv6n\") pod \"test-curl-7725-8a419a\" (UID: \"06c18733-eea4-42ce-96a0-0d3816ef029e\") " pod="opendatahub/test-curl-7725-8a419a"
Sep 15 21:46:16.175295 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:16.175254    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgv6n\" (UniqueName: \"kubernetes.io/projected/06c18733-eea4-42ce-96a0-0d3816ef029e-kube-api-access-wgv6n\") pod \"test-curl-7725-8a419a\" (UID: \"06c18733-eea4-42ce-96a0-0d3816ef029e\") " pod="opendatahub/test-curl-7725-8a419a"
Sep 15 21:46:16.196657 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:46:16.196573    2575 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7725-c9e4c7" container="test-curl-7725-c9e4c7"
Sep 15 21:46:16.250421 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:16.250380    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-8a419a"
Sep 15 21:46:16.527645 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:16.527612    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7725-8a419a"]
Sep 15 21:46:16.529392 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:46:16.529365    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06c18733_eea4_42ce_96a0_0d3816ef029e.slice/crio-fd2196eeb1ae1a9f31a92b58592cec3b27b57f50df1e2e786b6dd33aac51b6d0 WatchSource:0}: Error finding container fd2196eeb1ae1a9f31a92b58592cec3b27b57f50df1e2e786b6dd33aac51b6d0: Status 404 returned error can't find the container with id fd2196eeb1ae1a9f31a92b58592cec3b27b57f50df1e2e786b6dd33aac51b6d0
Sep 15 21:46:17.205428 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:17.205388    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7725-8a419a" event={"ID":"06c18733-eea4-42ce-96a0-0d3816ef029e","Type":"ContainerStarted","Data":"24668c4b1dc297c260f70a8368fcb5efe6d3f882a9e4d462403575046c6e09cc"}
Sep 15 21:46:17.205930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:17.205441    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7725-8a419a" event={"ID":"06c18733-eea4-42ce-96a0-0d3816ef029e","Type":"ContainerStarted","Data":"fd2196eeb1ae1a9f31a92b58592cec3b27b57f50df1e2e786b6dd33aac51b6d0"}
Sep 15 21:46:17.205930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:17.205496    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7725-c9e4c7" podUID="1b26f70c-1b8e-4a9d-ab57-c6d515ec320d" containerName="test-curl-7725-c9e4c7" containerID="cri-o://1ac6e3b1317dac89b0eab6481778b9b8cfe2c8f8cd52afb0d444953089cd36f0" gracePeriod=2
Sep 15 21:46:17.221448 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:17.221385    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7725-8a419a" podStartSLOduration=2.221365441 podStartE2EDuration="2.221365441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:46:17.219019107 +0000 UTC m=+1215.031027549" watchObservedRunningTime="2026-09-15 21:46:17.221365441 +0000 UTC m=+1215.033373885"
Sep 15 21:46:17.344689 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:17.344654    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-c9bb9c"
Sep 15 21:46:17.482623 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:17.482519    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2z8w\" (UniqueName: \"kubernetes.io/projected/5a459ad0-26bd-4271-ba2b-ddd586d1e6a9-kube-api-access-w2z8w\") pod \"5a459ad0-26bd-4271-ba2b-ddd586d1e6a9\" (UID: \"5a459ad0-26bd-4271-ba2b-ddd586d1e6a9\") "
Sep 15 21:46:17.484881 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:17.484843    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a459ad0-26bd-4271-ba2b-ddd586d1e6a9-kube-api-access-w2z8w" (OuterVolumeSpecName: "kube-api-access-w2z8w") pod "5a459ad0-26bd-4271-ba2b-ddd586d1e6a9" (UID: "5a459ad0-26bd-4271-ba2b-ddd586d1e6a9"). InnerVolumeSpecName "kube-api-access-w2z8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:17.584270 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:17.584228    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2z8w\" (UniqueName: \"kubernetes.io/projected/5a459ad0-26bd-4271-ba2b-ddd586d1e6a9-kube-api-access-w2z8w\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:46:17.863545 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:17.863510    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7725-8a419a"]
Sep 15 21:46:17.867539 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:17.867505    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7725-8a419a"]
Sep 15 21:46:17.982456 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:17.982417    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7725-01bbda"]
Sep 15 21:46:17.982979 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:17.982964    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06c18733-eea4-42ce-96a0-0d3816ef029e" containerName="test-curl-7725-8a419a"
Sep 15 21:46:17.982979 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:17.982980    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="06c18733-eea4-42ce-96a0-0d3816ef029e" containerName="test-curl-7725-8a419a"
Sep 15 21:46:17.983100 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:17.983055    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="06c18733-eea4-42ce-96a0-0d3816ef029e" containerName="test-curl-7725-8a419a"
Sep 15 21:46:17.986310 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:17.986292    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-01bbda"
Sep 15 21:46:17.991648 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:17.991619    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7725-01bbda"]
Sep 15 21:46:18.090405 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:18.090365    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzg7q\" (UniqueName: \"kubernetes.io/projected/204825a7-bcda-4b32-8f8b-af8f8d1bf3a5-kube-api-access-wzg7q\") pod \"test-curl-7725-01bbda\" (UID: \"204825a7-bcda-4b32-8f8b-af8f8d1bf3a5\") " pod="opendatahub/test-curl-7725-01bbda"
Sep 15 21:46:18.191158 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:18.191055    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzg7q\" (UniqueName: \"kubernetes.io/projected/204825a7-bcda-4b32-8f8b-af8f8d1bf3a5-kube-api-access-wzg7q\") pod \"test-curl-7725-01bbda\" (UID: \"204825a7-bcda-4b32-8f8b-af8f8d1bf3a5\") " pod="opendatahub/test-curl-7725-01bbda"
Sep 15 21:46:18.198996 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:18.198968    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzg7q\" (UniqueName: \"kubernetes.io/projected/204825a7-bcda-4b32-8f8b-af8f8d1bf3a5-kube-api-access-wzg7q\") pod \"test-curl-7725-01bbda\" (UID: \"204825a7-bcda-4b32-8f8b-af8f8d1bf3a5\") " pod="opendatahub/test-curl-7725-01bbda"
Sep 15 21:46:18.213109 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:18.213072    2575 generic.go:356] "Generic (PLEG): container finished" podID="5a459ad0-26bd-4271-ba2b-ddd586d1e6a9" containerID="32e698ec2def8c7181b38a02d6cfa2fbb8114662c8ec5c4813cd402b2a42ff34" exitCode=137
Sep 15 21:46:18.213518 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:18.213128    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-c9bb9c"
Sep 15 21:46:18.213518 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:18.213182    2575 scope.go:117] "RemoveContainer" containerID="32e698ec2def8c7181b38a02d6cfa2fbb8114662c8ec5c4813cd402b2a42ff34"
Sep 15 21:46:18.213650 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:46:18.213539    2575 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7725-8a419a" container="test-curl-7725-8a419a"
Sep 15 21:46:18.223723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:18.223700    2575 scope.go:117] "RemoveContainer" containerID="32e698ec2def8c7181b38a02d6cfa2fbb8114662c8ec5c4813cd402b2a42ff34"
Sep 15 21:46:18.224163 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:46:18.224140    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e698ec2def8c7181b38a02d6cfa2fbb8114662c8ec5c4813cd402b2a42ff34\": container with ID starting with 32e698ec2def8c7181b38a02d6cfa2fbb8114662c8ec5c4813cd402b2a42ff34 not found: ID does not exist" containerID="32e698ec2def8c7181b38a02d6cfa2fbb8114662c8ec5c4813cd402b2a42ff34"
Sep 15 21:46:18.224215 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:18.224173    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e698ec2def8c7181b38a02d6cfa2fbb8114662c8ec5c4813cd402b2a42ff34"} err="failed to get container status \"32e698ec2def8c7181b38a02d6cfa2fbb8114662c8ec5c4813cd402b2a42ff34\": rpc error: code = NotFound desc = could not find container \"32e698ec2def8c7181b38a02d6cfa2fbb8114662c8ec5c4813cd402b2a42ff34\": container with ID starting with 32e698ec2def8c7181b38a02d6cfa2fbb8114662c8ec5c4813cd402b2a42ff34 not found: ID does not exist"
Sep 15 21:46:18.224540 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:46:18.224520    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7725-c9bb9c\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="5a459ad0-26bd-4271-ba2b-ddd586d1e6a9" pod="opendatahub/test-curl-7725-c9bb9c"
Sep 15 21:46:18.298025 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:18.297991    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-01bbda"
Sep 15 21:46:18.570125 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:18.570097    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7725-01bbda"]
Sep 15 21:46:18.572261 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:46:18.572231    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod204825a7_bcda_4b32_8f8b_af8f8d1bf3a5.slice/crio-f5203faf55a8822a518ceca6f06e3181dac882244e8247d7ea3a7a0af246bd0e WatchSource:0}: Error finding container f5203faf55a8822a518ceca6f06e3181dac882244e8247d7ea3a7a0af246bd0e: Status 404 returned error can't find the container with id f5203faf55a8822a518ceca6f06e3181dac882244e8247d7ea3a7a0af246bd0e
Sep 15 21:46:18.748013 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:18.747968    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a459ad0-26bd-4271-ba2b-ddd586d1e6a9" path="/var/lib/kubelet/pods/5a459ad0-26bd-4271-ba2b-ddd586d1e6a9/volumes"
Sep 15 21:46:19.222198 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:19.222163    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7725-01bbda" event={"ID":"204825a7-bcda-4b32-8f8b-af8f8d1bf3a5","Type":"ContainerStarted","Data":"8889eac5c87ce15f64e88588c0e3df915e146d8b74ee8587db707b4532aeacc1"}
Sep 15 21:46:19.222556 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:19.222213    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7725-01bbda" event={"ID":"204825a7-bcda-4b32-8f8b-af8f8d1bf3a5","Type":"ContainerStarted","Data":"f5203faf55a8822a518ceca6f06e3181dac882244e8247d7ea3a7a0af246bd0e"}
Sep 15 21:46:19.222556 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:19.222310    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7725-8a419a" podUID="06c18733-eea4-42ce-96a0-0d3816ef029e" containerName="test-curl-7725-8a419a" containerID="cri-o://24668c4b1dc297c260f70a8368fcb5efe6d3f882a9e4d462403575046c6e09cc" gracePeriod=2
Sep 15 21:46:19.236352 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:19.236290    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7725-01bbda" podStartSLOduration=2.236270664 podStartE2EDuration="2.236270664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:46:19.234893263 +0000 UTC m=+1217.046901707" watchObservedRunningTime="2026-09-15 21:46:19.236270664 +0000 UTC m=+1217.048279107"
Sep 15 21:46:19.394161 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:19.394135    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-c9e4c7"
Sep 15 21:46:19.507414 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:19.507307    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrjql\" (UniqueName: \"kubernetes.io/projected/1b26f70c-1b8e-4a9d-ab57-c6d515ec320d-kube-api-access-nrjql\") pod \"1b26f70c-1b8e-4a9d-ab57-c6d515ec320d\" (UID: \"1b26f70c-1b8e-4a9d-ab57-c6d515ec320d\") "
Sep 15 21:46:19.509644 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:19.509616    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b26f70c-1b8e-4a9d-ab57-c6d515ec320d-kube-api-access-nrjql" (OuterVolumeSpecName: "kube-api-access-nrjql") pod "1b26f70c-1b8e-4a9d-ab57-c6d515ec320d" (UID: "1b26f70c-1b8e-4a9d-ab57-c6d515ec320d"). InnerVolumeSpecName "kube-api-access-nrjql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:19.609074 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:19.609028    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrjql\" (UniqueName: \"kubernetes.io/projected/1b26f70c-1b8e-4a9d-ab57-c6d515ec320d-kube-api-access-nrjql\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:46:19.862735 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:19.862690    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7725-01bbda"]
Sep 15 21:46:19.866085 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:19.866045    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7725-01bbda"]
Sep 15 21:46:20.230455 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:20.230362    2575 generic.go:356] "Generic (PLEG): container finished" podID="1b26f70c-1b8e-4a9d-ab57-c6d515ec320d" containerID="1ac6e3b1317dac89b0eab6481778b9b8cfe2c8f8cd52afb0d444953089cd36f0" exitCode=137
Sep 15 21:46:20.230455 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:20.230407    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-c9e4c7"
Sep 15 21:46:20.230974 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:20.230460    2575 scope.go:117] "RemoveContainer" containerID="1ac6e3b1317dac89b0eab6481778b9b8cfe2c8f8cd52afb0d444953089cd36f0"
Sep 15 21:46:20.230974 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:46:20.230781    2575 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7725-01bbda" container="test-curl-7725-01bbda"
Sep 15 21:46:20.240361 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:46:20.240332    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7725-c9e4c7\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="1b26f70c-1b8e-4a9d-ab57-c6d515ec320d" pod="opendatahub/test-curl-7725-c9e4c7"
Sep 15 21:46:20.240805 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:20.240787    2575 scope.go:117] "RemoveContainer" containerID="1ac6e3b1317dac89b0eab6481778b9b8cfe2c8f8cd52afb0d444953089cd36f0"
Sep 15 21:46:20.241132 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:46:20.241114    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac6e3b1317dac89b0eab6481778b9b8cfe2c8f8cd52afb0d444953089cd36f0\": container with ID starting with 1ac6e3b1317dac89b0eab6481778b9b8cfe2c8f8cd52afb0d444953089cd36f0 not found: ID does not exist" containerID="1ac6e3b1317dac89b0eab6481778b9b8cfe2c8f8cd52afb0d444953089cd36f0"
Sep 15 21:46:20.241208 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:20.241138    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac6e3b1317dac89b0eab6481778b9b8cfe2c8f8cd52afb0d444953089cd36f0"} err="failed to get container status \"1ac6e3b1317dac89b0eab6481778b9b8cfe2c8f8cd52afb0d444953089cd36f0\": rpc error: code = NotFound desc = could not find container \"1ac6e3b1317dac89b0eab6481778b9b8cfe2c8f8cd52afb0d444953089cd36f0\": container with ID starting with 1ac6e3b1317dac89b0eab6481778b9b8cfe2c8f8cd52afb0d444953089cd36f0 not found: ID does not exist"
Sep 15 21:46:20.748690 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:20.748654    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b26f70c-1b8e-4a9d-ab57-c6d515ec320d" path="/var/lib/kubelet/pods/1b26f70c-1b8e-4a9d-ab57-c6d515ec320d/volumes"
Sep 15 21:46:21.237966 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:21.237924    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7725-01bbda" podUID="204825a7-bcda-4b32-8f8b-af8f8d1bf3a5" containerName="test-curl-7725-01bbda" containerID="cri-o://8889eac5c87ce15f64e88588c0e3df915e146d8b74ee8587db707b4532aeacc1" gracePeriod=2
Sep 15 21:46:21.401117 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:21.401092    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-8a419a"
Sep 15 21:46:21.532466 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:21.532368    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgv6n\" (UniqueName: \"kubernetes.io/projected/06c18733-eea4-42ce-96a0-0d3816ef029e-kube-api-access-wgv6n\") pod \"06c18733-eea4-42ce-96a0-0d3816ef029e\" (UID: \"06c18733-eea4-42ce-96a0-0d3816ef029e\") "
Sep 15 21:46:21.534654 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:21.534619    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c18733-eea4-42ce-96a0-0d3816ef029e-kube-api-access-wgv6n" (OuterVolumeSpecName: "kube-api-access-wgv6n") pod "06c18733-eea4-42ce-96a0-0d3816ef029e" (UID: "06c18733-eea4-42ce-96a0-0d3816ef029e"). InnerVolumeSpecName "kube-api-access-wgv6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:21.633927 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:21.633886    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgv6n\" (UniqueName: \"kubernetes.io/projected/06c18733-eea4-42ce-96a0-0d3816ef029e-kube-api-access-wgv6n\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:46:22.245063 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:22.245024    2575 generic.go:356] "Generic (PLEG): container finished" podID="06c18733-eea4-42ce-96a0-0d3816ef029e" containerID="24668c4b1dc297c260f70a8368fcb5efe6d3f882a9e4d462403575046c6e09cc" exitCode=137
Sep 15 21:46:22.245063 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:22.245066    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-8a419a"
Sep 15 21:46:22.245548 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:22.245115    2575 scope.go:117] "RemoveContainer" containerID="24668c4b1dc297c260f70a8368fcb5efe6d3f882a9e4d462403575046c6e09cc"
Sep 15 21:46:22.255224 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:46:22.255191    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7725-8a419a\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="06c18733-eea4-42ce-96a0-0d3816ef029e" pod="opendatahub/test-curl-7725-8a419a"
Sep 15 21:46:22.255318 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:22.255257    2575 scope.go:117] "RemoveContainer" containerID="24668c4b1dc297c260f70a8368fcb5efe6d3f882a9e4d462403575046c6e09cc"
Sep 15 21:46:22.255626 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:46:22.255608    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24668c4b1dc297c260f70a8368fcb5efe6d3f882a9e4d462403575046c6e09cc\": container with ID starting with 24668c4b1dc297c260f70a8368fcb5efe6d3f882a9e4d462403575046c6e09cc not found: ID does not exist" containerID="24668c4b1dc297c260f70a8368fcb5efe6d3f882a9e4d462403575046c6e09cc"
Sep 15 21:46:22.255705 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:22.255641    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24668c4b1dc297c260f70a8368fcb5efe6d3f882a9e4d462403575046c6e09cc"} err="failed to get container status \"24668c4b1dc297c260f70a8368fcb5efe6d3f882a9e4d462403575046c6e09cc\": rpc error: code = NotFound desc = could not find container \"24668c4b1dc297c260f70a8368fcb5efe6d3f882a9e4d462403575046c6e09cc\": container with ID starting with 24668c4b1dc297c260f70a8368fcb5efe6d3f882a9e4d462403575046c6e09cc not found: ID does not exist"
Sep 15 21:46:22.752565 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:46:22.752523    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7725-8a419a\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="06c18733-eea4-42ce-96a0-0d3816ef029e" pod="opendatahub/test-curl-7725-8a419a"
Sep 15 21:46:22.756694 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:22.756662    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06c18733-eea4-42ce-96a0-0d3816ef029e" path="/var/lib/kubelet/pods/06c18733-eea4-42ce-96a0-0d3816ef029e/volumes"
Sep 15 21:46:23.420950 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:23.420924    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-01bbda"
Sep 15 21:46:23.554590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:23.554493    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzg7q\" (UniqueName: \"kubernetes.io/projected/204825a7-bcda-4b32-8f8b-af8f8d1bf3a5-kube-api-access-wzg7q\") pod \"204825a7-bcda-4b32-8f8b-af8f8d1bf3a5\" (UID: \"204825a7-bcda-4b32-8f8b-af8f8d1bf3a5\") "
Sep 15 21:46:23.556772 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:23.556724    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204825a7-bcda-4b32-8f8b-af8f8d1bf3a5-kube-api-access-wzg7q" (OuterVolumeSpecName: "kube-api-access-wzg7q") pod "204825a7-bcda-4b32-8f8b-af8f8d1bf3a5" (UID: "204825a7-bcda-4b32-8f8b-af8f8d1bf3a5"). InnerVolumeSpecName "kube-api-access-wzg7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:46:23.656145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:23.656108    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzg7q\" (UniqueName: \"kubernetes.io/projected/204825a7-bcda-4b32-8f8b-af8f8d1bf3a5-kube-api-access-wzg7q\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:46:24.259830 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:24.259793    2575 generic.go:356] "Generic (PLEG): container finished" podID="204825a7-bcda-4b32-8f8b-af8f8d1bf3a5" containerID="8889eac5c87ce15f64e88588c0e3df915e146d8b74ee8587db707b4532aeacc1" exitCode=137
Sep 15 21:46:24.260015 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:24.259878    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7725-01bbda"
Sep 15 21:46:24.260015 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:24.259934    2575 scope.go:117] "RemoveContainer" containerID="8889eac5c87ce15f64e88588c0e3df915e146d8b74ee8587db707b4532aeacc1"
Sep 15 21:46:24.271213 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:46:24.271030    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7725-01bbda\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="204825a7-bcda-4b32-8f8b-af8f8d1bf3a5" pod="opendatahub/test-curl-7725-01bbda"
Sep 15 21:46:24.273096 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:24.272952    2575 scope.go:117] "RemoveContainer" containerID="8889eac5c87ce15f64e88588c0e3df915e146d8b74ee8587db707b4532aeacc1"
Sep 15 21:46:24.273342 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:46:24.273318    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8889eac5c87ce15f64e88588c0e3df915e146d8b74ee8587db707b4532aeacc1\": container with ID starting with 8889eac5c87ce15f64e88588c0e3df915e146d8b74ee8587db707b4532aeacc1 not found: ID does not exist" containerID="8889eac5c87ce15f64e88588c0e3df915e146d8b74ee8587db707b4532aeacc1"
Sep 15 21:46:24.273402 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:24.273352    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8889eac5c87ce15f64e88588c0e3df915e146d8b74ee8587db707b4532aeacc1"} err="failed to get container status \"8889eac5c87ce15f64e88588c0e3df915e146d8b74ee8587db707b4532aeacc1\": rpc error: code = NotFound desc = could not find container \"8889eac5c87ce15f64e88588c0e3df915e146d8b74ee8587db707b4532aeacc1\": container with ID starting with 8889eac5c87ce15f64e88588c0e3df915e146d8b74ee8587db707b4532aeacc1 not found: ID does not exist"
Sep 15 21:46:24.748503 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:24.748462    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="204825a7-bcda-4b32-8f8b-af8f8d1bf3a5" path="/var/lib/kubelet/pods/204825a7-bcda-4b32-8f8b-af8f8d1bf3a5/volumes"
Sep 15 21:46:29.284974 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.284939    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"]
Sep 15 21:46:29.285480 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.285465    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="204825a7-bcda-4b32-8f8b-af8f8d1bf3a5" containerName="test-curl-7725-01bbda"
Sep 15 21:46:29.285524 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.285482    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="204825a7-bcda-4b32-8f8b-af8f8d1bf3a5" containerName="test-curl-7725-01bbda"
Sep 15 21:46:29.285578 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.285566    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="204825a7-bcda-4b32-8f8b-af8f8d1bf3a5" containerName="test-curl-7725-01bbda"
Sep 15 21:46:29.291641 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.291615    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"
Sep 15 21:46:29.294734 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.294699    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-459b305b\""
Sep 15 21:46:29.294924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.294784    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-459b305b-dockercfg-mkgbp\""
Sep 15 21:46:29.301285 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.301253    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"]
Sep 15 21:46:29.313683 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.313645    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sxg8\" (UniqueName: \"kubernetes.io/projected/137c6f2d-c929-4db7-aa12-047afeba8525-kube-api-access-9sxg8\") pod \"payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d\" (UID: \"137c6f2d-c929-4db7-aa12-047afeba8525\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"
Sep 15 21:46:29.313904 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.313821    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137c6f2d-c929-4db7-aa12-047afeba8525-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d\" (UID: \"137c6f2d-c929-4db7-aa12-047afeba8525\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"
Sep 15 21:46:29.335819 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.335782    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"]
Sep 15 21:46:29.339670 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.339647    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"
Sep 15 21:46:29.362433 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.362396    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"]
Sep 15 21:46:29.415185 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.415145    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441c308e-f33d-46c6-8f80-dce21fab23b2-plugins-config-volume\") pod \"payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl\" (UID: \"441c308e-f33d-46c6-8f80-dce21fab23b2\") " pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"
Sep 15 21:46:29.415368 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.415195    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137c6f2d-c929-4db7-aa12-047afeba8525-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d\" (UID: \"137c6f2d-c929-4db7-aa12-047afeba8525\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"
Sep 15 21:46:29.415368 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.415283    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxkqb\" (UniqueName: \"kubernetes.io/projected/441c308e-f33d-46c6-8f80-dce21fab23b2-kube-api-access-mxkqb\") pod \"payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl\" (UID: \"441c308e-f33d-46c6-8f80-dce21fab23b2\") " pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"
Sep 15 21:46:29.415437 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.415394    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sxg8\" (UniqueName: \"kubernetes.io/projected/137c6f2d-c929-4db7-aa12-047afeba8525-kube-api-access-9sxg8\") pod \"payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d\" (UID: \"137c6f2d-c929-4db7-aa12-047afeba8525\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"
Sep 15 21:46:29.415883 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.415865    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137c6f2d-c929-4db7-aa12-047afeba8525-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d\" (UID: \"137c6f2d-c929-4db7-aa12-047afeba8525\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"
Sep 15 21:46:29.429415 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.429388    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sxg8\" (UniqueName: \"kubernetes.io/projected/137c6f2d-c929-4db7-aa12-047afeba8525-kube-api-access-9sxg8\") pod \"payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d\" (UID: \"137c6f2d-c929-4db7-aa12-047afeba8525\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"
Sep 15 21:46:29.516773 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.516722    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441c308e-f33d-46c6-8f80-dce21fab23b2-plugins-config-volume\") pod \"payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl\" (UID: \"441c308e-f33d-46c6-8f80-dce21fab23b2\") " pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"
Sep 15 21:46:29.516973 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.516846    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxkqb\" (UniqueName: \"kubernetes.io/projected/441c308e-f33d-46c6-8f80-dce21fab23b2-kube-api-access-mxkqb\") pod \"payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl\" (UID: \"441c308e-f33d-46c6-8f80-dce21fab23b2\") " pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"
Sep 15 21:46:29.517440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.517418    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441c308e-f33d-46c6-8f80-dce21fab23b2-plugins-config-volume\") pod \"payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl\" (UID: \"441c308e-f33d-46c6-8f80-dce21fab23b2\") " pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"
Sep 15 21:46:29.524557 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.524530    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxkqb\" (UniqueName: \"kubernetes.io/projected/441c308e-f33d-46c6-8f80-dce21fab23b2-kube-api-access-mxkqb\") pod \"payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl\" (UID: \"441c308e-f33d-46c6-8f80-dce21fab23b2\") " pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"
Sep 15 21:46:29.604264 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.604225    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"
Sep 15 21:46:29.650977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.650941    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"
Sep 15 21:46:29.880366 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.880336    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"]
Sep 15 21:46:29.882939 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:46:29.882916    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137c6f2d_c929_4db7_aa12_047afeba8525.slice/crio-2dad6cb60a9c69d7cacc6fb1ca2b46c39689cacea702d2058ca74568cb61fb3c WatchSource:0}: Error finding container 2dad6cb60a9c69d7cacc6fb1ca2b46c39689cacea702d2058ca74568cb61fb3c: Status 404 returned error can't find the container with id 2dad6cb60a9c69d7cacc6fb1ca2b46c39689cacea702d2058ca74568cb61fb3c
Sep 15 21:46:29.938041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:29.938017    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"]
Sep 15 21:46:29.940519 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:46:29.940486    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441c308e_f33d_46c6_8f80_dce21fab23b2.slice/crio-056262bd256eedf774907619e565bc1370c83ae657eea3988b53ca5ee2fb9445 WatchSource:0}: Error finding container 056262bd256eedf774907619e565bc1370c83ae657eea3988b53ca5ee2fb9445: Status 404 returned error can't find the container with id 056262bd256eedf774907619e565bc1370c83ae657eea3988b53ca5ee2fb9445
Sep 15 21:46:30.301783 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:30.301646    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d" event={"ID":"137c6f2d-c929-4db7-aa12-047afeba8525","Type":"ContainerStarted","Data":"a171a50314e8eb90962828f2248bcf1995e9bae0438be423d271eca0b643eafc"}
Sep 15 21:46:30.301783 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:30.301725    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d" event={"ID":"137c6f2d-c929-4db7-aa12-047afeba8525","Type":"ContainerStarted","Data":"2dad6cb60a9c69d7cacc6fb1ca2b46c39689cacea702d2058ca74568cb61fb3c"}
Sep 15 21:46:30.302297 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:30.301877    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"
Sep 15 21:46:30.303628 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:30.303600    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl" event={"ID":"441c308e-f33d-46c6-8f80-dce21fab23b2","Type":"ContainerStarted","Data":"809b9fa3afcf8878825b4b23917d3939472e9d64efeb2f7de1df3fadb12c8f70"}
Sep 15 21:46:30.303813 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:30.303646    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl" event={"ID":"441c308e-f33d-46c6-8f80-dce21fab23b2","Type":"ContainerStarted","Data":"056262bd256eedf774907619e565bc1370c83ae657eea3988b53ca5ee2fb9445"}
Sep 15 21:46:30.303813 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:30.303729    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"
Sep 15 21:46:30.317519 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:30.317460    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d" podStartSLOduration=1.317434671 podStartE2EDuration="1.317434671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:46:30.316683378 +0000 UTC m=+1228.128691823" watchObservedRunningTime="2026-09-15 21:46:30.317434671 +0000 UTC m=+1228.129443117"
Sep 15 21:46:30.332259 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:30.332209    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl" podStartSLOduration=1.33218879 podStartE2EDuration="1.33218879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:46:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:46:30.331358953 +0000 UTC m=+1228.143367397" watchObservedRunningTime="2026-09-15 21:46:30.33218879 +0000 UTC m=+1228.144197234"
Sep 15 21:46:39.837202 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:39.837169    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:46:41.312296 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:41.312263    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"
Sep 15 21:46:41.312679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:46:41.312340    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"
Sep 15 21:47:03.565642 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:03.565557    2575 scope.go:117] "RemoveContainer" containerID="0b1e32f1ef822cf65f3de3bf4744f09f327c9029988f7f468e17fa2912f56cb4"
Sep 15 21:47:05.234908 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:05.234860    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:47:16.813423 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:16.813385    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"]
Sep 15 21:47:16.830320 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:47:16.830285    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-self-signed-certs: secret "test-model-1af1db-kserve-self-signed-certs" not found
Sep 15 21:47:16.830489 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:47:16.830361    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs podName:10dac161-67f3-44ab-b402-092ed04cf803 nodeName:}" failed. No retries permitted until 2026-09-15 21:47:17.330343208 +0000 UTC m=+1275.142351630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs") pod "test-model-1af1db-kserve-68f79b48f9-fj2qk" (UID: "10dac161-67f3-44ab-b402-092ed04cf803") : secret "test-model-1af1db-kserve-self-signed-certs" not found
Sep 15 21:47:17.336739 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:47:17.336697    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-self-signed-certs: secret "test-model-1af1db-kserve-self-signed-certs" not found
Sep 15 21:47:17.336739 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:47:17.336818    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs podName:10dac161-67f3-44ab-b402-092ed04cf803 nodeName:}" failed. No retries permitted until 2026-09-15 21:47:18.336797071 +0000 UTC m=+1276.148805493 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs") pod "test-model-1af1db-kserve-68f79b48f9-fj2qk" (UID: "10dac161-67f3-44ab-b402-092ed04cf803") : secret "test-model-1af1db-kserve-self-signed-certs" not found
Sep 15 21:47:18.347574 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:47:18.347539    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-self-signed-certs: secret "test-model-1af1db-kserve-self-signed-certs" not found
Sep 15 21:47:18.348006 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:47:18.347615    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs podName:10dac161-67f3-44ab-b402-092ed04cf803 nodeName:}" failed. No retries permitted until 2026-09-15 21:47:20.347600612 +0000 UTC m=+1278.159609034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs") pod "test-model-1af1db-kserve-68f79b48f9-fj2qk" (UID: "10dac161-67f3-44ab-b402-092ed04cf803") : secret "test-model-1af1db-kserve-self-signed-certs" not found
Sep 15 21:47:20.369007 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:47:20.368962    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-self-signed-certs: secret "test-model-1af1db-kserve-self-signed-certs" not found
Sep 15 21:47:20.369418 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:47:20.369052    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs podName:10dac161-67f3-44ab-b402-092ed04cf803 nodeName:}" failed. No retries permitted until 2026-09-15 21:47:24.369033291 +0000 UTC m=+1282.181041719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs") pod "test-model-1af1db-kserve-68f79b48f9-fj2qk" (UID: "10dac161-67f3-44ab-b402-092ed04cf803") : secret "test-model-1af1db-kserve-self-signed-certs" not found
Sep 15 21:47:24.410094 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:47:24.410054    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-self-signed-certs: secret "test-model-1af1db-kserve-self-signed-certs" not found
Sep 15 21:47:24.410525 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:47:24.410141    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs podName:10dac161-67f3-44ab-b402-092ed04cf803 nodeName:}" failed. No retries permitted until 2026-09-15 21:47:32.410125571 +0000 UTC m=+1290.222133996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs") pod "test-model-1af1db-kserve-68f79b48f9-fj2qk" (UID: "10dac161-67f3-44ab-b402-092ed04cf803") : secret "test-model-1af1db-kserve-self-signed-certs" not found
Sep 15 21:47:29.636573 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:29.636537    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:47:31.073928 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.073891    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1af1db-29825145-qs5fq"]
Sep 15 21:47:31.079486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.079455    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1af1db-29825145-qs5fq"]
Sep 15 21:47:31.115846 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.115806    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"]
Sep 15 21:47:31.116123 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.116068    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl" podUID="c8299e83-f37c-4ba8-8e14-f5f67691e191" containerName="payload-pre-processing" containerID="cri-o://1ae774baff3d9fba698c9b57a13c96e865c59ebcd5d8f3a7994a46ce9d305806" gracePeriod=30
Sep 15 21:47:31.378208 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.378175    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"
Sep 15 21:47:31.489721 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.489684    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmlbn\" (UniqueName: \"kubernetes.io/projected/c8299e83-f37c-4ba8-8e14-f5f67691e191-kube-api-access-jmlbn\") pod \"c8299e83-f37c-4ba8-8e14-f5f67691e191\" (UID: \"c8299e83-f37c-4ba8-8e14-f5f67691e191\") "
Sep 15 21:47:31.489940 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.489796    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8299e83-f37c-4ba8-8e14-f5f67691e191-plugins-config-volume\") pod \"c8299e83-f37c-4ba8-8e14-f5f67691e191\" (UID: \"c8299e83-f37c-4ba8-8e14-f5f67691e191\") "
Sep 15 21:47:31.490174 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.490135    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8299e83-f37c-4ba8-8e14-f5f67691e191-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c8299e83-f37c-4ba8-8e14-f5f67691e191" (UID: "c8299e83-f37c-4ba8-8e14-f5f67691e191"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:47:31.492070 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.492043    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8299e83-f37c-4ba8-8e14-f5f67691e191-kube-api-access-jmlbn" (OuterVolumeSpecName: "kube-api-access-jmlbn") pod "c8299e83-f37c-4ba8-8e14-f5f67691e191" (UID: "c8299e83-f37c-4ba8-8e14-f5f67691e191"). InnerVolumeSpecName "kube-api-access-jmlbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:47:31.591088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.591041    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmlbn\" (UniqueName: \"kubernetes.io/projected/c8299e83-f37c-4ba8-8e14-f5f67691e191-kube-api-access-jmlbn\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:47:31.591088 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.591078    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8299e83-f37c-4ba8-8e14-f5f67691e191-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:47:31.693252 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.693154    2575 generic.go:356] "Generic (PLEG): container finished" podID="c8299e83-f37c-4ba8-8e14-f5f67691e191" containerID="1ae774baff3d9fba698c9b57a13c96e865c59ebcd5d8f3a7994a46ce9d305806" exitCode=0
Sep 15 21:47:31.693252 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.693202    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"
Sep 15 21:47:31.693252 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.693242    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl" event={"ID":"c8299e83-f37c-4ba8-8e14-f5f67691e191","Type":"ContainerDied","Data":"1ae774baff3d9fba698c9b57a13c96e865c59ebcd5d8f3a7994a46ce9d305806"}
Sep 15 21:47:31.693498 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.693295    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl" event={"ID":"c8299e83-f37c-4ba8-8e14-f5f67691e191","Type":"ContainerDied","Data":"b559deb19c9ae7b9f53b4c4555d7317748f70466cd36b0b1e33776c75dc5d97f"}
Sep 15 21:47:31.693498 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.693318    2575 scope.go:117] "RemoveContainer" containerID="1ae774baff3d9fba698c9b57a13c96e865c59ebcd5d8f3a7994a46ce9d305806"
Sep 15 21:47:31.703624 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.703605    2575 scope.go:117] "RemoveContainer" containerID="1ae774baff3d9fba698c9b57a13c96e865c59ebcd5d8f3a7994a46ce9d305806"
Sep 15 21:47:31.704044 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:47:31.704016    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae774baff3d9fba698c9b57a13c96e865c59ebcd5d8f3a7994a46ce9d305806\": container with ID starting with 1ae774baff3d9fba698c9b57a13c96e865c59ebcd5d8f3a7994a46ce9d305806 not found: ID does not exist" containerID="1ae774baff3d9fba698c9b57a13c96e865c59ebcd5d8f3a7994a46ce9d305806"
Sep 15 21:47:31.704138 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.704054    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae774baff3d9fba698c9b57a13c96e865c59ebcd5d8f3a7994a46ce9d305806"} err="failed to get container status \"1ae774baff3d9fba698c9b57a13c96e865c59ebcd5d8f3a7994a46ce9d305806\": rpc error: code = NotFound desc = could not find container \"1ae774baff3d9fba698c9b57a13c96e865c59ebcd5d8f3a7994a46ce9d305806\": container with ID starting with 1ae774baff3d9fba698c9b57a13c96e865c59ebcd5d8f3a7994a46ce9d305806 not found: ID does not exist"
Sep 15 21:47:31.713987 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.713953    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"]
Sep 15 21:47:31.718739 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:31.718708    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-1af1db-56cd65c7f8-s4thl"]
Sep 15 21:47:32.501843 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:47:32.501802    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-self-signed-certs: secret "test-model-1af1db-kserve-self-signed-certs" not found
Sep 15 21:47:32.502235 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:47:32.501875    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs podName:10dac161-67f3-44ab-b402-092ed04cf803 nodeName:}" failed. No retries permitted until 2026-09-15 21:47:48.501859362 +0000 UTC m=+1306.313867797 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs") pod "test-model-1af1db-kserve-68f79b48f9-fj2qk" (UID: "10dac161-67f3-44ab-b402-092ed04cf803") : secret "test-model-1af1db-kserve-self-signed-certs" not found
Sep 15 21:47:32.621937 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:32.621893    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk" podUID="10dac161-67f3-44ab-b402-092ed04cf803" containerName="main" containerID="cri-o://7e4cb359b66f42c195a66500b21e6af41d18c60bba03f1019f31f0e488fcb630" gracePeriod=45
Sep 15 21:47:32.749071 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:32.749037    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8766399-c40c-453a-94d3-137282e231ca" path="/var/lib/kubelet/pods/a8766399-c40c-453a-94d3-137282e231ca/volumes"
Sep 15 21:47:32.749412 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:32.749399    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8299e83-f37c-4ba8-8e14-f5f67691e191" path="/var/lib/kubelet/pods/c8299e83-f37c-4ba8-8e14-f5f67691e191/volumes"
Sep 15 21:47:32.881663 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:32.881634    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:47:33.008138 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.008109    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs\") pod \"10dac161-67f3-44ab-b402-092ed04cf803\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") "
Sep 15 21:47:33.008325 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.008179    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8fsz\" (UniqueName: \"kubernetes.io/projected/10dac161-67f3-44ab-b402-092ed04cf803-kube-api-access-l8fsz\") pod \"10dac161-67f3-44ab-b402-092ed04cf803\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") "
Sep 15 21:47:33.008325 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.008246    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-tmp-dir\") pod \"10dac161-67f3-44ab-b402-092ed04cf803\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") "
Sep 15 21:47:33.008325 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.008284    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-home\") pod \"10dac161-67f3-44ab-b402-092ed04cf803\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") "
Sep 15 21:47:33.008325 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.008310    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-model-cache\") pod \"10dac161-67f3-44ab-b402-092ed04cf803\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") "
Sep 15 21:47:33.008522 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.008337    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-dshm\") pod \"10dac161-67f3-44ab-b402-092ed04cf803\" (UID: \"10dac161-67f3-44ab-b402-092ed04cf803\") "
Sep 15 21:47:33.008594 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.008568    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-home" (OuterVolumeSpecName: "home") pod "10dac161-67f3-44ab-b402-092ed04cf803" (UID: "10dac161-67f3-44ab-b402-092ed04cf803"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:47:33.008664 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.008557    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10dac161-67f3-44ab-b402-092ed04cf803" (UID: "10dac161-67f3-44ab-b402-092ed04cf803"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:47:33.008664 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.008599    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-model-cache" (OuterVolumeSpecName: "model-cache") pod "10dac161-67f3-44ab-b402-092ed04cf803" (UID: "10dac161-67f3-44ab-b402-092ed04cf803"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:47:33.008997 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.008975    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:47:33.008997 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.008998    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:47:33.009171 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.009013    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:47:33.010703 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.010680    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10dac161-67f3-44ab-b402-092ed04cf803" (UID: "10dac161-67f3-44ab-b402-092ed04cf803"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:47:33.010703 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.010688    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10dac161-67f3-44ab-b402-092ed04cf803-kube-api-access-l8fsz" (OuterVolumeSpecName: "kube-api-access-l8fsz") pod "10dac161-67f3-44ab-b402-092ed04cf803" (UID: "10dac161-67f3-44ab-b402-092ed04cf803"). InnerVolumeSpecName "kube-api-access-l8fsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:47:33.010863 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.010721    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-dshm" (OuterVolumeSpecName: "dshm") pod "10dac161-67f3-44ab-b402-092ed04cf803" (UID: "10dac161-67f3-44ab-b402-092ed04cf803"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:47:33.110377 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.110341    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8fsz\" (UniqueName: \"kubernetes.io/projected/10dac161-67f3-44ab-b402-092ed04cf803-kube-api-access-l8fsz\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:47:33.110377 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.110372    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10dac161-67f3-44ab-b402-092ed04cf803-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:47:33.110377 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.110383    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10dac161-67f3-44ab-b402-092ed04cf803-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:47:33.707406 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.707353    2575 generic.go:356] "Generic (PLEG): container finished" podID="10dac161-67f3-44ab-b402-092ed04cf803" containerID="7e4cb359b66f42c195a66500b21e6af41d18c60bba03f1019f31f0e488fcb630" exitCode=0
Sep 15 21:47:33.707930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.707424    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"
Sep 15 21:47:33.707930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.707440    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk" event={"ID":"10dac161-67f3-44ab-b402-092ed04cf803","Type":"ContainerDied","Data":"7e4cb359b66f42c195a66500b21e6af41d18c60bba03f1019f31f0e488fcb630"}
Sep 15 21:47:33.707930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.707489    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk" event={"ID":"10dac161-67f3-44ab-b402-092ed04cf803","Type":"ContainerDied","Data":"2f06508d6362e7fa2ed997ad7c25067b4ac3e904b6a252d782f03ef6803f9d93"}
Sep 15 21:47:33.707930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.707508    2575 scope.go:117] "RemoveContainer" containerID="7e4cb359b66f42c195a66500b21e6af41d18c60bba03f1019f31f0e488fcb630"
Sep 15 21:47:33.718202 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.718182    2575 scope.go:117] "RemoveContainer" containerID="7e4cb359b66f42c195a66500b21e6af41d18c60bba03f1019f31f0e488fcb630"
Sep 15 21:47:33.718547 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:47:33.718523    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4cb359b66f42c195a66500b21e6af41d18c60bba03f1019f31f0e488fcb630\": container with ID starting with 7e4cb359b66f42c195a66500b21e6af41d18c60bba03f1019f31f0e488fcb630 not found: ID does not exist" containerID="7e4cb359b66f42c195a66500b21e6af41d18c60bba03f1019f31f0e488fcb630"
Sep 15 21:47:33.718652 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.718554    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4cb359b66f42c195a66500b21e6af41d18c60bba03f1019f31f0e488fcb630"} err="failed to get container status \"7e4cb359b66f42c195a66500b21e6af41d18c60bba03f1019f31f0e488fcb630\": rpc error: code = NotFound desc = could not find container \"7e4cb359b66f42c195a66500b21e6af41d18c60bba03f1019f31f0e488fcb630\": container with ID starting with 7e4cb359b66f42c195a66500b21e6af41d18c60bba03f1019f31f0e488fcb630 not found: ID does not exist"
Sep 15 21:47:33.728946 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.728900    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"]
Sep 15 21:47:33.732257 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:33.732228    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1af1db/test-model-1af1db-kserve-68f79b48f9-fj2qk"]
Sep 15 21:47:34.748391 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:34.748349    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10dac161-67f3-44ab-b402-092ed04cf803" path="/var/lib/kubelet/pods/10dac161-67f3-44ab-b402-092ed04cf803/volumes"
Sep 15 21:47:39.331869 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.331830    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"]
Sep 15 21:47:39.332436 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.332413    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8299e83-f37c-4ba8-8e14-f5f67691e191" containerName="payload-pre-processing"
Sep 15 21:47:39.332436 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.332435    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8299e83-f37c-4ba8-8e14-f5f67691e191" containerName="payload-pre-processing"
Sep 15 21:47:39.332653 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.332467    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10dac161-67f3-44ab-b402-092ed04cf803" containerName="main"
Sep 15 21:47:39.332653 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.332474    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="10dac161-67f3-44ab-b402-092ed04cf803" containerName="main"
Sep 15 21:47:39.332653 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.332559    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="10dac161-67f3-44ab-b402-092ed04cf803" containerName="main"
Sep 15 21:47:39.332653 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.332569    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8299e83-f37c-4ba8-8e14-f5f67691e191" containerName="payload-pre-processing"
Sep 15 21:47:39.337160 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.337139    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.339535 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.339510    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-459b305b\"/\"kube-root-ca.crt\""
Sep 15 21:47:39.340332 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.340308    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-459b305b\"/\"default-dockercfg-pzknq\""
Sep 15 21:47:39.340470 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.340347    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-459b305b\"/\"e2e-lifecycle-model-459b305b-kserve-self-signed-certs\""
Sep 15 21:47:39.340561 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.340349    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-459b305b\"/\"openshift-service-ca.crt\""
Sep 15 21:47:39.345045 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.345019    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"]
Sep 15 21:47:39.479653 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.479607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-dshm\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.479857 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.479665    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-tmp-dir\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.479857 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.479746    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2f3adb-386e-4a67-8859-16bff8ef3928-tls-certs\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.479857 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.479848    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-model-cache\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.480014 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.479940    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7gbt\" (UniqueName: \"kubernetes.io/projected/2b2f3adb-386e-4a67-8859-16bff8ef3928-kube-api-access-f7gbt\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.480014 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.479972    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-home\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.581501 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.581458    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7gbt\" (UniqueName: \"kubernetes.io/projected/2b2f3adb-386e-4a67-8859-16bff8ef3928-kube-api-access-f7gbt\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.581501 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.581506    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-home\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.581747 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.581549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-dshm\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.581747 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.581579    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-tmp-dir\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.581747 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.581624    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2f3adb-386e-4a67-8859-16bff8ef3928-tls-certs\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.581747 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.581670    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-model-cache\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.582099 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.582044    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-tmp-dir\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.582166 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.582100    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-home\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.582166 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.582147    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-model-cache\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.584136 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.584102    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-dshm\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.584331 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.584312    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2f3adb-386e-4a67-8859-16bff8ef3928-tls-certs\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.588709 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.588682    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7gbt\" (UniqueName: \"kubernetes.io/projected/2b2f3adb-386e-4a67-8859-16bff8ef3928-kube-api-access-f7gbt\") pod \"e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") " pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.649482 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.649444    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:39.935230 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:39.935202    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"]
Sep 15 21:47:39.937034 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:47:39.936993    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b2f3adb_386e_4a67_8859_16bff8ef3928.slice/crio-879106e53c6443e1179e20a7b0d51d272a4e91b02e0b8480f72389c0a61e3217 WatchSource:0}: Error finding container 879106e53c6443e1179e20a7b0d51d272a4e91b02e0b8480f72389c0a61e3217: Status 404 returned error can't find the container with id 879106e53c6443e1179e20a7b0d51d272a4e91b02e0b8480f72389c0a61e3217
Sep 15 21:47:40.751890 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:40.751847    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b" event={"ID":"2b2f3adb-386e-4a67-8859-16bff8ef3928","Type":"ContainerStarted","Data":"bfa09b8a2e51d4aff3f5691957e51b30551b50b29f8bdb5158cd921fa2816107"}
Sep 15 21:47:40.751890 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:40.751895    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b" event={"ID":"2b2f3adb-386e-4a67-8859-16bff8ef3928","Type":"ContainerStarted","Data":"879106e53c6443e1179e20a7b0d51d272a4e91b02e0b8480f72389c0a61e3217"}
Sep 15 21:47:40.768204 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:40.768105    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b" podStartSLOduration=1.768084304 podStartE2EDuration="1.768084304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:47:40.767361283 +0000 UTC m=+1298.579369764" watchObservedRunningTime="2026-09-15 21:47:40.768084304 +0000 UTC m=+1298.580092749"
Sep 15 21:47:48.381736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.381692    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2d9c4-29825145-cwl78"]
Sep 15 21:47:48.385069 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.385038    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a2d9c4-29825145-cwl78"]
Sep 15 21:47:48.415245 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.415212    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"]
Sep 15 21:47:48.415489 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.415435    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq" podUID="3682aab2-51bd-41bf-adbd-ac92d55fc492" containerName="payload-pre-processing" containerID="cri-o://4ecd9f9476ead038159c6a94aff4424ba5250422a739d7fe868ecc8d042f5a63" gracePeriod=30
Sep 15 21:47:48.675331 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.675303    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"
Sep 15 21:47:48.748789 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.748720    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e29a21a3-b861-45fd-87f3-9e0651d80823" path="/var/lib/kubelet/pods/e29a21a3-b861-45fd-87f3-9e0651d80823/volumes"
Sep 15 21:47:48.772857 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.772820    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpx8c\" (UniqueName: \"kubernetes.io/projected/3682aab2-51bd-41bf-adbd-ac92d55fc492-kube-api-access-qpx8c\") pod \"3682aab2-51bd-41bf-adbd-ac92d55fc492\" (UID: \"3682aab2-51bd-41bf-adbd-ac92d55fc492\") "
Sep 15 21:47:48.773060 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.772939    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3682aab2-51bd-41bf-adbd-ac92d55fc492-plugins-config-volume\") pod \"3682aab2-51bd-41bf-adbd-ac92d55fc492\" (UID: \"3682aab2-51bd-41bf-adbd-ac92d55fc492\") "
Sep 15 21:47:48.773315 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.773290    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3682aab2-51bd-41bf-adbd-ac92d55fc492-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3682aab2-51bd-41bf-adbd-ac92d55fc492" (UID: "3682aab2-51bd-41bf-adbd-ac92d55fc492"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:47:48.775207 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.775183    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3682aab2-51bd-41bf-adbd-ac92d55fc492-kube-api-access-qpx8c" (OuterVolumeSpecName: "kube-api-access-qpx8c") pod "3682aab2-51bd-41bf-adbd-ac92d55fc492" (UID: "3682aab2-51bd-41bf-adbd-ac92d55fc492"). InnerVolumeSpecName "kube-api-access-qpx8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:47:48.800049 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.800009    2575 generic.go:356] "Generic (PLEG): container finished" podID="3682aab2-51bd-41bf-adbd-ac92d55fc492" containerID="4ecd9f9476ead038159c6a94aff4424ba5250422a739d7fe868ecc8d042f5a63" exitCode=0
Sep 15 21:47:48.800218 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.800061    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq" event={"ID":"3682aab2-51bd-41bf-adbd-ac92d55fc492","Type":"ContainerDied","Data":"4ecd9f9476ead038159c6a94aff4424ba5250422a739d7fe868ecc8d042f5a63"}
Sep 15 21:47:48.800218 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.800075    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"
Sep 15 21:47:48.800218 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.800103    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq" event={"ID":"3682aab2-51bd-41bf-adbd-ac92d55fc492","Type":"ContainerDied","Data":"99e3fe5122a2cf29e6615ee6e5aa0549b6aa3ad8bc18ed16dc229bbed3f8ee78"}
Sep 15 21:47:48.800218 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.800124    2575 scope.go:117] "RemoveContainer" containerID="4ecd9f9476ead038159c6a94aff4424ba5250422a739d7fe868ecc8d042f5a63"
Sep 15 21:47:48.810966 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.810948    2575 scope.go:117] "RemoveContainer" containerID="4ecd9f9476ead038159c6a94aff4424ba5250422a739d7fe868ecc8d042f5a63"
Sep 15 21:47:48.811313 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:47:48.811292    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ecd9f9476ead038159c6a94aff4424ba5250422a739d7fe868ecc8d042f5a63\": container with ID starting with 4ecd9f9476ead038159c6a94aff4424ba5250422a739d7fe868ecc8d042f5a63 not found: ID does not exist" containerID="4ecd9f9476ead038159c6a94aff4424ba5250422a739d7fe868ecc8d042f5a63"
Sep 15 21:47:48.811369 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.811322    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ecd9f9476ead038159c6a94aff4424ba5250422a739d7fe868ecc8d042f5a63"} err="failed to get container status \"4ecd9f9476ead038159c6a94aff4424ba5250422a739d7fe868ecc8d042f5a63\": rpc error: code = NotFound desc = could not find container \"4ecd9f9476ead038159c6a94aff4424ba5250422a739d7fe868ecc8d042f5a63\": container with ID starting with 4ecd9f9476ead038159c6a94aff4424ba5250422a739d7fe868ecc8d042f5a63 not found: ID does not exist"
Sep 15 21:47:48.822015 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.821984    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"]
Sep 15 21:47:48.824170 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.824149    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a2d9c4-7bbd5698c4-gd9bq"]
Sep 15 21:47:48.874198 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.874162    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpx8c\" (UniqueName: \"kubernetes.io/projected/3682aab2-51bd-41bf-adbd-ac92d55fc492-kube-api-access-qpx8c\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:47:48.874198 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:48.874191    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3682aab2-51bd-41bf-adbd-ac92d55fc492-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:47:49.649896 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:49.649861    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:49.650377 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:49.649917    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:49.663050 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:49.663024    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:49.818779 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:49.818735    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:47:50.748046 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:47:50.748014    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3682aab2-51bd-41bf-adbd-ac92d55fc492" path="/var/lib/kubelet/pods/3682aab2-51bd-41bf-adbd-ac92d55fc492/volumes"
Sep 15 21:48:03.608437 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:03.608404    2575 scope.go:117] "RemoveContainer" containerID="8e842d95c7c2808f5f4f5fec9d73ecd1d79ff3e26c5f2d8babe2a25c76137158"
Sep 15 21:48:03.617784 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:03.617738    2575 scope.go:117] "RemoveContainer" containerID="4eda1a2ca5f833412830df5d4e7afacd813dc3ca846b23533ac36cfae99de7bc"
Sep 15 21:48:04.630410 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:04.630375    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:48:28.135350 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.135261    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"]
Sep 15 21:48:28.135836 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.135768    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3682aab2-51bd-41bf-adbd-ac92d55fc492" containerName="payload-pre-processing"
Sep 15 21:48:28.135836 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.135781    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3682aab2-51bd-41bf-adbd-ac92d55fc492" containerName="payload-pre-processing"
Sep 15 21:48:28.135922 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.135872    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3682aab2-51bd-41bf-adbd-ac92d55fc492" containerName="payload-pre-processing"
Sep 15 21:48:28.140648 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.140625    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.142838 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.142810    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-56e305\"/\"e2e-conflict-b-d70fe03c-kserve-self-signed-certs\""
Sep 15 21:48:28.149720 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.149689    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"]
Sep 15 21:48:28.285603 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.285559    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7jp4\" (UniqueName: \"kubernetes.io/projected/3e3392b1-85bd-4001-bd6c-5d757331cc19-kube-api-access-t7jp4\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.285603 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.285607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-model-cache\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.285876 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.285668    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-dshm\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.285876 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.285689    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-tmp-dir\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.285876 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.285786    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-home\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.285876 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.285832    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3392b1-85bd-4001-bd6c-5d757331cc19-tls-certs\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.386651 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.386546    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7jp4\" (UniqueName: \"kubernetes.io/projected/3e3392b1-85bd-4001-bd6c-5d757331cc19-kube-api-access-t7jp4\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.386651 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.386588    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-model-cache\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.386651 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.386639    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-dshm\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.386996 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.386660    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-tmp-dir\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.386996 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.386680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-home\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.386996 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.386712    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3392b1-85bd-4001-bd6c-5d757331cc19-tls-certs\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.387145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.387105    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-home\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.387210 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.387177    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-tmp-dir\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.387274 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.387256    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-model-cache\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.389212 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.389190    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-dshm\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.389380 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.389363    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3392b1-85bd-4001-bd6c-5d757331cc19-tls-certs\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.394152 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.394126    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7jp4\" (UniqueName: \"kubernetes.io/projected/3e3392b1-85bd-4001-bd6c-5d757331cc19-kube-api-access-t7jp4\") pod \"e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") " pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.452501 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.452463    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:28.726170 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:28.726138    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"]
Sep 15 21:48:28.728572 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:48:28.728542    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e3392b1_85bd_4001_bd6c_5d757331cc19.slice/crio-2f24641b7d4d00e1583b79120f0f9694b5cf8fc97e998b34fd8606646c04f139 WatchSource:0}: Error finding container 2f24641b7d4d00e1583b79120f0f9694b5cf8fc97e998b34fd8606646c04f139: Status 404 returned error can't find the container with id 2f24641b7d4d00e1583b79120f0f9694b5cf8fc97e998b34fd8606646c04f139
Sep 15 21:48:29.038380 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:29.038276    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59" event={"ID":"3e3392b1-85bd-4001-bd6c-5d757331cc19","Type":"ContainerStarted","Data":"560361cce5792dd3804264eb18ed88887ad05c94ebbdda6ac77c93a654a08ac6"}
Sep 15 21:48:29.038380 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:29.038330    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59" event={"ID":"3e3392b1-85bd-4001-bd6c-5d757331cc19","Type":"ContainerStarted","Data":"2f24641b7d4d00e1583b79120f0f9694b5cf8fc97e998b34fd8606646c04f139"}
Sep 15 21:48:29.055181 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:29.055124    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59" podStartSLOduration=1.055104599 podStartE2EDuration="1.055104599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:48:29.053018902 +0000 UTC m=+1346.865027351" watchObservedRunningTime="2026-09-15 21:48:29.055104599 +0000 UTC m=+1346.867113086"
Sep 15 21:48:32.234413 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:32.234367    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:48:38.442849 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:38.442810    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"]
Sep 15 21:48:38.446728 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:38.446702    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"
Sep 15 21:48:38.449082 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:38.449022    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e7a648\""
Sep 15 21:48:38.449221 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:38.449155    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e7a648-dockercfg-m5mzc\""
Sep 15 21:48:38.452718 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:38.452693    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:38.452925 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:38.452741    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:38.456549 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:38.456522    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"]
Sep 15 21:48:38.466810 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:38.466779    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:38.489255 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:38.489216    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5zcs\" (UniqueName: \"kubernetes.io/projected/f433221e-a8c8-407c-9eaa-549526cdb6f0-kube-api-access-b5zcs\") pod \"payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k\" (UID: \"f433221e-a8c8-407c-9eaa-549526cdb6f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"
Sep 15 21:48:38.489486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:38.489471    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f433221e-a8c8-407c-9eaa-549526cdb6f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k\" (UID: \"f433221e-a8c8-407c-9eaa-549526cdb6f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"
Sep 15 21:48:38.590198 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:38.590156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f433221e-a8c8-407c-9eaa-549526cdb6f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k\" (UID: \"f433221e-a8c8-407c-9eaa-549526cdb6f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"
Sep 15 21:48:38.590427 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:38.590224    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5zcs\" (UniqueName: \"kubernetes.io/projected/f433221e-a8c8-407c-9eaa-549526cdb6f0-kube-api-access-b5zcs\") pod \"payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k\" (UID: \"f433221e-a8c8-407c-9eaa-549526cdb6f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"
Sep 15 21:48:38.590970 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:38.590948    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f433221e-a8c8-407c-9eaa-549526cdb6f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k\" (UID: \"f433221e-a8c8-407c-9eaa-549526cdb6f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"
Sep 15 21:48:38.597652 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:38.597630    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5zcs\" (UniqueName: \"kubernetes.io/projected/f433221e-a8c8-407c-9eaa-549526cdb6f0-kube-api-access-b5zcs\") pod \"payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k\" (UID: \"f433221e-a8c8-407c-9eaa-549526cdb6f0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"
Sep 15 21:48:38.759746 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:38.759659    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"
Sep 15 21:48:39.034589 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:39.034563    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"]
Sep 15 21:48:39.037119 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:48:39.037089    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf433221e_a8c8_407c_9eaa_549526cdb6f0.slice/crio-5bd65bc7fb25d152ed99df20401078eadc25bf0cab77b57112a6089c649ca9f8 WatchSource:0}: Error finding container 5bd65bc7fb25d152ed99df20401078eadc25bf0cab77b57112a6089c649ca9f8: Status 404 returned error can't find the container with id 5bd65bc7fb25d152ed99df20401078eadc25bf0cab77b57112a6089c649ca9f8
Sep 15 21:48:39.097580 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:39.097548    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k" event={"ID":"f433221e-a8c8-407c-9eaa-549526cdb6f0","Type":"ContainerStarted","Data":"5bd65bc7fb25d152ed99df20401078eadc25bf0cab77b57112a6089c649ca9f8"}
Sep 15 21:48:39.109974 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:39.109950    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:48:40.105354 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:40.105318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k" event={"ID":"f433221e-a8c8-407c-9eaa-549526cdb6f0","Type":"ContainerStarted","Data":"561b1d977224b1ac54b2d251ce176196b594879a339d9b3d2302339359f9acab"}
Sep 15 21:48:40.121706 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:40.121651    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k" podStartSLOduration=2.121630811 podStartE2EDuration="2.121630811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:48:40.120088979 +0000 UTC m=+1357.932097436" watchObservedRunningTime="2026-09-15 21:48:40.121630811 +0000 UTC m=+1357.933639255"
Sep 15 21:48:41.111243 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:41.111217    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"
Sep 15 21:48:52.118730 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:48:52.118696    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"
Sep 15 21:49:03.656906 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:03.656872    2575 scope.go:117] "RemoveContainer" containerID="c44cb25299cab0c021782025f374d04c6d3d09dcccc0fae2b30fa76b030fd0e6"
Sep 15 21:49:08.535365 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:08.535331    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:49:36.700070 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:36.700033    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-459b305b-qppph"]
Sep 15 21:49:36.704004 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:36.703982    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-459b305b-qppph"
Sep 15 21:49:36.706015 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:36.705992    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:49:36.706148 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:36.706068    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 21:49:36.710433 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:36.710408    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-459b305b-qppph"]
Sep 15 21:49:36.805539 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:36.805503    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2c60405-86ab-4edd-8044-7ff6cdcb4416-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-459b305b-qppph\" (UID: \"b2c60405-86ab-4edd-8044-7ff6cdcb4416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-459b305b-qppph"
Sep 15 21:49:36.906658 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:36.906623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2c60405-86ab-4edd-8044-7ff6cdcb4416-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-459b305b-qppph\" (UID: \"b2c60405-86ab-4edd-8044-7ff6cdcb4416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-459b305b-qppph"
Sep 15 21:49:36.907352 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:36.907335    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2c60405-86ab-4edd-8044-7ff6cdcb4416-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-459b305b-qppph\" (UID: \"b2c60405-86ab-4edd-8044-7ff6cdcb4416\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-459b305b-qppph"
Sep 15 21:49:37.016897 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:37.016808    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-459b305b-qppph"
Sep 15 21:49:37.291981 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:37.291885    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-459b305b-qppph"]
Sep 15 21:49:37.295330 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:49:37.295302    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c60405_86ab_4edd_8044_7ff6cdcb4416.slice/crio-acc54b8d5a4fc878c1d41de3e0f697704a2d2acfe5fd51ea1877676b51221971 WatchSource:0}: Error finding container acc54b8d5a4fc878c1d41de3e0f697704a2d2acfe5fd51ea1877676b51221971: Status 404 returned error can't find the container with id acc54b8d5a4fc878c1d41de3e0f697704a2d2acfe5fd51ea1877676b51221971
Sep 15 21:49:37.434193 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:37.434155    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-459b305b-qppph" event={"ID":"b2c60405-86ab-4edd-8044-7ff6cdcb4416","Type":"ContainerStarted","Data":"acc54b8d5a4fc878c1d41de3e0f697704a2d2acfe5fd51ea1877676b51221971"}
Sep 15 21:49:38.442119 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:38.442077    2575 generic.go:356] "Generic (PLEG): container finished" podID="b2c60405-86ab-4edd-8044-7ff6cdcb4416" containerID="176cb69b0734cc3644df70258107fa06477543c7f9861faefd1dae7c8bda67c1" exitCode=0
Sep 15 21:49:38.442483 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:38.442168    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-459b305b-qppph" event={"ID":"b2c60405-86ab-4edd-8044-7ff6cdcb4416","Type":"ContainerDied","Data":"176cb69b0734cc3644df70258107fa06477543c7f9861faefd1dae7c8bda67c1"}
Sep 15 21:49:39.604180 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:39.604137    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-459b305b-qppph"
Sep 15 21:49:39.635789 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:39.635729    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2c60405-86ab-4edd-8044-7ff6cdcb4416-maas-api-service-ca\") pod \"b2c60405-86ab-4edd-8044-7ff6cdcb4416\" (UID: \"b2c60405-86ab-4edd-8044-7ff6cdcb4416\") "
Sep 15 21:49:39.636109 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:39.636086    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c60405-86ab-4edd-8044-7ff6cdcb4416-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2c60405-86ab-4edd-8044-7ff6cdcb4416" (UID: "b2c60405-86ab-4edd-8044-7ff6cdcb4416"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:49:39.636196 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:39.636182    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2c60405-86ab-4edd-8044-7ff6cdcb4416-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:49:40.456500 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:40.456470    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-459b305b-qppph"
Sep 15 21:49:40.456691 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:40.456470    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-459b305b-qppph" event={"ID":"b2c60405-86ab-4edd-8044-7ff6cdcb4416","Type":"ContainerDied","Data":"acc54b8d5a4fc878c1d41de3e0f697704a2d2acfe5fd51ea1877676b51221971"}
Sep 15 21:49:40.456691 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:40.456606    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acc54b8d5a4fc878c1d41de3e0f697704a2d2acfe5fd51ea1877676b51221971"
Sep 15 21:49:51.132809 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:51.132765    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:49:52.213783 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.213716    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"]
Sep 15 21:49:52.214273 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.214011    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl" podUID="441c308e-f33d-46c6-8f80-dce21fab23b2" containerName="payload-processing" containerID="cri-o://809b9fa3afcf8878825b4b23917d3939472e9d64efeb2f7de1df3fadb12c8f70" gracePeriod=30
Sep 15 21:49:52.221608 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.220992    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"]
Sep 15 21:49:52.221608 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.221224    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d" podUID="137c6f2d-c929-4db7-aa12-047afeba8525" containerName="payload-pre-processing" containerID="cri-o://a171a50314e8eb90962828f2248bcf1995e9bae0438be423d271eca0b643eafc" gracePeriod=30
Sep 15 21:49:52.371091 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.371050    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-459b305b-qppph"]
Sep 15 21:49:52.373462 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.373425    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-459b305b-qppph"]
Sep 15 21:49:52.518924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.518898    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"
Sep 15 21:49:52.523848 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.523821    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"
Sep 15 21:49:52.528282 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.528253    2575 generic.go:356] "Generic (PLEG): container finished" podID="441c308e-f33d-46c6-8f80-dce21fab23b2" containerID="809b9fa3afcf8878825b4b23917d3939472e9d64efeb2f7de1df3fadb12c8f70" exitCode=0
Sep 15 21:49:52.528465 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.528295    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"
Sep 15 21:49:52.528465 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.528325    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl" event={"ID":"441c308e-f33d-46c6-8f80-dce21fab23b2","Type":"ContainerDied","Data":"809b9fa3afcf8878825b4b23917d3939472e9d64efeb2f7de1df3fadb12c8f70"}
Sep 15 21:49:52.528465 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.528377    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl" event={"ID":"441c308e-f33d-46c6-8f80-dce21fab23b2","Type":"ContainerDied","Data":"056262bd256eedf774907619e565bc1370c83ae657eea3988b53ca5ee2fb9445"}
Sep 15 21:49:52.528465 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.528400    2575 scope.go:117] "RemoveContainer" containerID="809b9fa3afcf8878825b4b23917d3939472e9d64efeb2f7de1df3fadb12c8f70"
Sep 15 21:49:52.529907 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.529880    2575 generic.go:356] "Generic (PLEG): container finished" podID="137c6f2d-c929-4db7-aa12-047afeba8525" containerID="a171a50314e8eb90962828f2248bcf1995e9bae0438be423d271eca0b643eafc" exitCode=0
Sep 15 21:49:52.530021 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.529914    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"
Sep 15 21:49:52.530021 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.529961    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d" event={"ID":"137c6f2d-c929-4db7-aa12-047afeba8525","Type":"ContainerDied","Data":"a171a50314e8eb90962828f2248bcf1995e9bae0438be423d271eca0b643eafc"}
Sep 15 21:49:52.530021 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.530002    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d" event={"ID":"137c6f2d-c929-4db7-aa12-047afeba8525","Type":"ContainerDied","Data":"2dad6cb60a9c69d7cacc6fb1ca2b46c39689cacea702d2058ca74568cb61fb3c"}
Sep 15 21:49:52.538952 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.538923    2575 scope.go:117] "RemoveContainer" containerID="809b9fa3afcf8878825b4b23917d3939472e9d64efeb2f7de1df3fadb12c8f70"
Sep 15 21:49:52.539424 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:49:52.539384    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809b9fa3afcf8878825b4b23917d3939472e9d64efeb2f7de1df3fadb12c8f70\": container with ID starting with 809b9fa3afcf8878825b4b23917d3939472e9d64efeb2f7de1df3fadb12c8f70 not found: ID does not exist" containerID="809b9fa3afcf8878825b4b23917d3939472e9d64efeb2f7de1df3fadb12c8f70"
Sep 15 21:49:52.539575 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.539429    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809b9fa3afcf8878825b4b23917d3939472e9d64efeb2f7de1df3fadb12c8f70"} err="failed to get container status \"809b9fa3afcf8878825b4b23917d3939472e9d64efeb2f7de1df3fadb12c8f70\": rpc error: code = NotFound desc = could not find container \"809b9fa3afcf8878825b4b23917d3939472e9d64efeb2f7de1df3fadb12c8f70\": container with ID starting with 809b9fa3afcf8878825b4b23917d3939472e9d64efeb2f7de1df3fadb12c8f70 not found: ID does not exist"
Sep 15 21:49:52.539575 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.539456    2575 scope.go:117] "RemoveContainer" containerID="a171a50314e8eb90962828f2248bcf1995e9bae0438be423d271eca0b643eafc"
Sep 15 21:49:52.552011 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.551942    2575 scope.go:117] "RemoveContainer" containerID="a171a50314e8eb90962828f2248bcf1995e9bae0438be423d271eca0b643eafc"
Sep 15 21:49:52.552436 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:49:52.552407    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a171a50314e8eb90962828f2248bcf1995e9bae0438be423d271eca0b643eafc\": container with ID starting with a171a50314e8eb90962828f2248bcf1995e9bae0438be423d271eca0b643eafc not found: ID does not exist" containerID="a171a50314e8eb90962828f2248bcf1995e9bae0438be423d271eca0b643eafc"
Sep 15 21:49:52.552551 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.552442    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a171a50314e8eb90962828f2248bcf1995e9bae0438be423d271eca0b643eafc"} err="failed to get container status \"a171a50314e8eb90962828f2248bcf1995e9bae0438be423d271eca0b643eafc\": rpc error: code = NotFound desc = could not find container \"a171a50314e8eb90962828f2248bcf1995e9bae0438be423d271eca0b643eafc\": container with ID starting with a171a50314e8eb90962828f2248bcf1995e9bae0438be423d271eca0b643eafc not found: ID does not exist"
Sep 15 21:49:52.577688 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.577653    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441c308e-f33d-46c6-8f80-dce21fab23b2-plugins-config-volume\") pod \"441c308e-f33d-46c6-8f80-dce21fab23b2\" (UID: \"441c308e-f33d-46c6-8f80-dce21fab23b2\") "
Sep 15 21:49:52.577894 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.577829    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxkqb\" (UniqueName: \"kubernetes.io/projected/441c308e-f33d-46c6-8f80-dce21fab23b2-kube-api-access-mxkqb\") pod \"441c308e-f33d-46c6-8f80-dce21fab23b2\" (UID: \"441c308e-f33d-46c6-8f80-dce21fab23b2\") "
Sep 15 21:49:52.577966 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.577920    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sxg8\" (UniqueName: \"kubernetes.io/projected/137c6f2d-c929-4db7-aa12-047afeba8525-kube-api-access-9sxg8\") pod \"137c6f2d-c929-4db7-aa12-047afeba8525\" (UID: \"137c6f2d-c929-4db7-aa12-047afeba8525\") "
Sep 15 21:49:52.577966 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.577959    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137c6f2d-c929-4db7-aa12-047afeba8525-plugins-config-volume\") pod \"137c6f2d-c929-4db7-aa12-047afeba8525\" (UID: \"137c6f2d-c929-4db7-aa12-047afeba8525\") "
Sep 15 21:49:52.578177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.578155    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/441c308e-f33d-46c6-8f80-dce21fab23b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "441c308e-f33d-46c6-8f80-dce21fab23b2" (UID: "441c308e-f33d-46c6-8f80-dce21fab23b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:49:52.578413 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.578384    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/137c6f2d-c929-4db7-aa12-047afeba8525-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "137c6f2d-c929-4db7-aa12-047afeba8525" (UID: "137c6f2d-c929-4db7-aa12-047afeba8525"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:49:52.578413 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.578395    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/441c308e-f33d-46c6-8f80-dce21fab23b2-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:49:52.580176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.580153    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/441c308e-f33d-46c6-8f80-dce21fab23b2-kube-api-access-mxkqb" (OuterVolumeSpecName: "kube-api-access-mxkqb") pod "441c308e-f33d-46c6-8f80-dce21fab23b2" (UID: "441c308e-f33d-46c6-8f80-dce21fab23b2"). InnerVolumeSpecName "kube-api-access-mxkqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:49:52.580284 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.580185    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137c6f2d-c929-4db7-aa12-047afeba8525-kube-api-access-9sxg8" (OuterVolumeSpecName: "kube-api-access-9sxg8") pod "137c6f2d-c929-4db7-aa12-047afeba8525" (UID: "137c6f2d-c929-4db7-aa12-047afeba8525"). InnerVolumeSpecName "kube-api-access-9sxg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:49:52.679270 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.679215    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxkqb\" (UniqueName: \"kubernetes.io/projected/441c308e-f33d-46c6-8f80-dce21fab23b2-kube-api-access-mxkqb\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:49:52.679270 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.679266    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sxg8\" (UniqueName: \"kubernetes.io/projected/137c6f2d-c929-4db7-aa12-047afeba8525-kube-api-access-9sxg8\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:49:52.679270 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.679277    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/137c6f2d-c929-4db7-aa12-047afeba8525-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:49:52.748423 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.748327    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2c60405-86ab-4edd-8044-7ff6cdcb4416" path="/var/lib/kubelet/pods/b2c60405-86ab-4edd-8044-7ff6cdcb4416/volumes"
Sep 15 21:49:52.845801 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.845742    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"]
Sep 15 21:49:52.851643 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.851609    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-459b305b-5499445cf7-jg2hl"]
Sep 15 21:49:52.860378 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.860344    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"]
Sep 15 21:49:52.865936 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:52.865898    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-459b305b-79655478bb-tzj9d"]
Sep 15 21:49:54.748320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:54.748231    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="137c6f2d-c929-4db7-aa12-047afeba8525" path="/var/lib/kubelet/pods/137c6f2d-c929-4db7-aa12-047afeba8525/volumes"
Sep 15 21:49:54.748688 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:49:54.748615    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="441c308e-f33d-46c6-8f80-dce21fab23b2" path="/var/lib/kubelet/pods/441c308e-f33d-46c6-8f80-dce21fab23b2/volumes"
Sep 15 21:50:00.346093 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:00.346052    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"]
Sep 15 21:50:15.368061 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.368000    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b" podUID="2b2f3adb-386e-4a67-8859-16bff8ef3928" containerName="main" containerID="cri-o://bfa09b8a2e51d4aff3f5691957e51b30551b50b29f8bdb5158cd921fa2816107" gracePeriod=45
Sep 15 21:50:15.629208 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.629129    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:50:15.673124 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.673090    2575 generic.go:356] "Generic (PLEG): container finished" podID="2b2f3adb-386e-4a67-8859-16bff8ef3928" containerID="bfa09b8a2e51d4aff3f5691957e51b30551b50b29f8bdb5158cd921fa2816107" exitCode=0
Sep 15 21:50:15.673304 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.673130    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b" event={"ID":"2b2f3adb-386e-4a67-8859-16bff8ef3928","Type":"ContainerDied","Data":"bfa09b8a2e51d4aff3f5691957e51b30551b50b29f8bdb5158cd921fa2816107"}
Sep 15 21:50:15.673304 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.673145    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"
Sep 15 21:50:15.673304 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.673160    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b" event={"ID":"2b2f3adb-386e-4a67-8859-16bff8ef3928","Type":"ContainerDied","Data":"879106e53c6443e1179e20a7b0d51d272a4e91b02e0b8480f72389c0a61e3217"}
Sep 15 21:50:15.673304 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.673176    2575 scope.go:117] "RemoveContainer" containerID="bfa09b8a2e51d4aff3f5691957e51b30551b50b29f8bdb5158cd921fa2816107"
Sep 15 21:50:15.684534 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.684499    2575 scope.go:117] "RemoveContainer" containerID="bfa09b8a2e51d4aff3f5691957e51b30551b50b29f8bdb5158cd921fa2816107"
Sep 15 21:50:15.684924 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:50:15.684903    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfa09b8a2e51d4aff3f5691957e51b30551b50b29f8bdb5158cd921fa2816107\": container with ID starting with bfa09b8a2e51d4aff3f5691957e51b30551b50b29f8bdb5158cd921fa2816107 not found: ID does not exist" containerID="bfa09b8a2e51d4aff3f5691957e51b30551b50b29f8bdb5158cd921fa2816107"
Sep 15 21:50:15.685008 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.684935    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfa09b8a2e51d4aff3f5691957e51b30551b50b29f8bdb5158cd921fa2816107"} err="failed to get container status \"bfa09b8a2e51d4aff3f5691957e51b30551b50b29f8bdb5158cd921fa2816107\": rpc error: code = NotFound desc = could not find container \"bfa09b8a2e51d4aff3f5691957e51b30551b50b29f8bdb5158cd921fa2816107\": container with ID starting with bfa09b8a2e51d4aff3f5691957e51b30551b50b29f8bdb5158cd921fa2816107 not found: ID does not exist"
Sep 15 21:50:15.730681 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.730641    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-home\") pod \"2b2f3adb-386e-4a67-8859-16bff8ef3928\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") "
Sep 15 21:50:15.730900 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.730705    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-model-cache\") pod \"2b2f3adb-386e-4a67-8859-16bff8ef3928\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") "
Sep 15 21:50:15.730900 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.730780    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-tmp-dir\") pod \"2b2f3adb-386e-4a67-8859-16bff8ef3928\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") "
Sep 15 21:50:15.730900 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.730813    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7gbt\" (UniqueName: \"kubernetes.io/projected/2b2f3adb-386e-4a67-8859-16bff8ef3928-kube-api-access-f7gbt\") pod \"2b2f3adb-386e-4a67-8859-16bff8ef3928\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") "
Sep 15 21:50:15.730900 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.730846    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2f3adb-386e-4a67-8859-16bff8ef3928-tls-certs\") pod \"2b2f3adb-386e-4a67-8859-16bff8ef3928\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") "
Sep 15 21:50:15.730900 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.730892    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-dshm\") pod \"2b2f3adb-386e-4a67-8859-16bff8ef3928\" (UID: \"2b2f3adb-386e-4a67-8859-16bff8ef3928\") "
Sep 15 21:50:15.731214 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.731026    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b2f3adb-386e-4a67-8859-16bff8ef3928" (UID: "2b2f3adb-386e-4a67-8859-16bff8ef3928"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:50:15.731214 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.731041    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-home" (OuterVolumeSpecName: "home") pod "2b2f3adb-386e-4a67-8859-16bff8ef3928" (UID: "2b2f3adb-386e-4a67-8859-16bff8ef3928"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:50:15.731214 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.731052    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b2f3adb-386e-4a67-8859-16bff8ef3928" (UID: "2b2f3adb-386e-4a67-8859-16bff8ef3928"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:50:15.731364 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.731252    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:15.731364 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.731265    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:15.731364 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.731274    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:15.733281 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.733255    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-dshm" (OuterVolumeSpecName: "dshm") pod "2b2f3adb-386e-4a67-8859-16bff8ef3928" (UID: "2b2f3adb-386e-4a67-8859-16bff8ef3928"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:50:15.733281 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.733256    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b2f3adb-386e-4a67-8859-16bff8ef3928-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b2f3adb-386e-4a67-8859-16bff8ef3928" (UID: "2b2f3adb-386e-4a67-8859-16bff8ef3928"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:50:15.733430 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.733296    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b2f3adb-386e-4a67-8859-16bff8ef3928-kube-api-access-f7gbt" (OuterVolumeSpecName: "kube-api-access-f7gbt") pod "2b2f3adb-386e-4a67-8859-16bff8ef3928" (UID: "2b2f3adb-386e-4a67-8859-16bff8ef3928"). InnerVolumeSpecName "kube-api-access-f7gbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:50:15.832201 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.832150    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7gbt\" (UniqueName: \"kubernetes.io/projected/2b2f3adb-386e-4a67-8859-16bff8ef3928-kube-api-access-f7gbt\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:15.832413 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.832281    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2f3adb-386e-4a67-8859-16bff8ef3928-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:15.832413 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.832311    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2f3adb-386e-4a67-8859-16bff8ef3928-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:15.994679 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.994646    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"]
Sep 15 21:50:15.996827 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:15.996800    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-459b305b/e2e-lifecycle-model-459b305b-kserve-69fc78c899-77n9b"]
Sep 15 21:50:16.748467 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:16.748430    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b2f3adb-386e-4a67-8859-16bff8ef3928" path="/var/lib/kubelet/pods/2b2f3adb-386e-4a67-8859-16bff8ef3928/volumes"
Sep 15 21:50:17.037595 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:17.037489    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"]
Sep 15 21:50:18.951564 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:18.951519    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"]
Sep 15 21:50:19.630151 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:19.630102    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:50:32.058902 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.058850    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59" podUID="3e3392b1-85bd-4001-bd6c-5d757331cc19" containerName="main" containerID="cri-o://560361cce5792dd3804264eb18ed88887ad05c94ebbdda6ac77c93a654a08ac6" gracePeriod=45
Sep 15 21:50:32.314813 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.314709    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:50:32.411700 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.411656    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-model-cache\") pod \"3e3392b1-85bd-4001-bd6c-5d757331cc19\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") "
Sep 15 21:50:32.411941 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.411725    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-tmp-dir\") pod \"3e3392b1-85bd-4001-bd6c-5d757331cc19\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") "
Sep 15 21:50:32.411941 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.411769    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-dshm\") pod \"3e3392b1-85bd-4001-bd6c-5d757331cc19\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") "
Sep 15 21:50:32.411941 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.411839    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-home\") pod \"3e3392b1-85bd-4001-bd6c-5d757331cc19\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") "
Sep 15 21:50:32.411941 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.411854    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3392b1-85bd-4001-bd6c-5d757331cc19-tls-certs\") pod \"3e3392b1-85bd-4001-bd6c-5d757331cc19\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") "
Sep 15 21:50:32.411941 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.411927    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7jp4\" (UniqueName: \"kubernetes.io/projected/3e3392b1-85bd-4001-bd6c-5d757331cc19-kube-api-access-t7jp4\") pod \"3e3392b1-85bd-4001-bd6c-5d757331cc19\" (UID: \"3e3392b1-85bd-4001-bd6c-5d757331cc19\") "
Sep 15 21:50:32.412193 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.412020    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e3392b1-85bd-4001-bd6c-5d757331cc19" (UID: "3e3392b1-85bd-4001-bd6c-5d757331cc19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:50:32.412193 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.412035    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e3392b1-85bd-4001-bd6c-5d757331cc19" (UID: "3e3392b1-85bd-4001-bd6c-5d757331cc19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:50:32.412193 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.412101    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-home" (OuterVolumeSpecName: "home") pod "3e3392b1-85bd-4001-bd6c-5d757331cc19" (UID: "3e3392b1-85bd-4001-bd6c-5d757331cc19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:50:32.412341 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.412244    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:32.412341 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.412258    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:32.412341 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.412268    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:32.414396 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.414365    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-dshm" (OuterVolumeSpecName: "dshm") pod "3e3392b1-85bd-4001-bd6c-5d757331cc19" (UID: "3e3392b1-85bd-4001-bd6c-5d757331cc19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:50:32.414396 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.414390    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e3392b1-85bd-4001-bd6c-5d757331cc19-kube-api-access-t7jp4" (OuterVolumeSpecName: "kube-api-access-t7jp4") pod "3e3392b1-85bd-4001-bd6c-5d757331cc19" (UID: "3e3392b1-85bd-4001-bd6c-5d757331cc19"). InnerVolumeSpecName "kube-api-access-t7jp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:50:32.414589 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.414407    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e3392b1-85bd-4001-bd6c-5d757331cc19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e3392b1-85bd-4001-bd6c-5d757331cc19" (UID: "3e3392b1-85bd-4001-bd6c-5d757331cc19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:50:32.513427 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.513390    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e3392b1-85bd-4001-bd6c-5d757331cc19-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:32.513427 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.513421    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7jp4\" (UniqueName: \"kubernetes.io/projected/3e3392b1-85bd-4001-bd6c-5d757331cc19-kube-api-access-t7jp4\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:32.513427 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.513430    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e3392b1-85bd-4001-bd6c-5d757331cc19-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:32.771730 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.771695    2575 generic.go:356] "Generic (PLEG): container finished" podID="3e3392b1-85bd-4001-bd6c-5d757331cc19" containerID="560361cce5792dd3804264eb18ed88887ad05c94ebbdda6ac77c93a654a08ac6" exitCode=0
Sep 15 21:50:32.771934 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.771773    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"
Sep 15 21:50:32.771934 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.771793    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59" event={"ID":"3e3392b1-85bd-4001-bd6c-5d757331cc19","Type":"ContainerDied","Data":"560361cce5792dd3804264eb18ed88887ad05c94ebbdda6ac77c93a654a08ac6"}
Sep 15 21:50:32.771934 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.771846    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59" event={"ID":"3e3392b1-85bd-4001-bd6c-5d757331cc19","Type":"ContainerDied","Data":"2f24641b7d4d00e1583b79120f0f9694b5cf8fc97e998b34fd8606646c04f139"}
Sep 15 21:50:32.771934 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.771867    2575 scope.go:117] "RemoveContainer" containerID="560361cce5792dd3804264eb18ed88887ad05c94ebbdda6ac77c93a654a08ac6"
Sep 15 21:50:32.781516 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.781495    2575 scope.go:117] "RemoveContainer" containerID="560361cce5792dd3804264eb18ed88887ad05c94ebbdda6ac77c93a654a08ac6"
Sep 15 21:50:32.781890 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:50:32.781865    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"560361cce5792dd3804264eb18ed88887ad05c94ebbdda6ac77c93a654a08ac6\": container with ID starting with 560361cce5792dd3804264eb18ed88887ad05c94ebbdda6ac77c93a654a08ac6 not found: ID does not exist" containerID="560361cce5792dd3804264eb18ed88887ad05c94ebbdda6ac77c93a654a08ac6"
Sep 15 21:50:32.782001 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.781897    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"560361cce5792dd3804264eb18ed88887ad05c94ebbdda6ac77c93a654a08ac6"} err="failed to get container status \"560361cce5792dd3804264eb18ed88887ad05c94ebbdda6ac77c93a654a08ac6\": rpc error: code = NotFound desc = could not find container \"560361cce5792dd3804264eb18ed88887ad05c94ebbdda6ac77c93a654a08ac6\": container with ID starting with 560361cce5792dd3804264eb18ed88887ad05c94ebbdda6ac77c93a654a08ac6 not found: ID does not exist"
Sep 15 21:50:32.789910 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.789875    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"]
Sep 15 21:50:32.792011 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:32.791989    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/e2e-conflict-b-d70fe03c-kserve-74c69bb9f-6bz59"]
Sep 15 21:50:33.972839 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:33.972786    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k" podUID="e285a82f-a400-4857-ad57-2edec5487452" containerName="main" containerID="cri-o://b47001eb1918d084b2e50830050879fc154f74e9ea5b0ff26116f234cb443c35" gracePeriod=45
Sep 15 21:50:34.236940 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.236864    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:50:34.330188 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.330142    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pgk6\" (UniqueName: \"kubernetes.io/projected/e285a82f-a400-4857-ad57-2edec5487452-kube-api-access-8pgk6\") pod \"e285a82f-a400-4857-ad57-2edec5487452\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") "
Sep 15 21:50:34.330188 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.330194    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-home\") pod \"e285a82f-a400-4857-ad57-2edec5487452\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") "
Sep 15 21:50:34.330396 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.330227    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e285a82f-a400-4857-ad57-2edec5487452-tls-certs\") pod \"e285a82f-a400-4857-ad57-2edec5487452\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") "
Sep 15 21:50:34.330396 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.330341    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-dshm\") pod \"e285a82f-a400-4857-ad57-2edec5487452\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") "
Sep 15 21:50:34.330396 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.330387    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-tmp-dir\") pod \"e285a82f-a400-4857-ad57-2edec5487452\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") "
Sep 15 21:50:34.330527 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.330417    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-model-cache\") pod \"e285a82f-a400-4857-ad57-2edec5487452\" (UID: \"e285a82f-a400-4857-ad57-2edec5487452\") "
Sep 15 21:50:34.330527 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.330488    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-home" (OuterVolumeSpecName: "home") pod "e285a82f-a400-4857-ad57-2edec5487452" (UID: "e285a82f-a400-4857-ad57-2edec5487452"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:50:34.330662 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.330637    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e285a82f-a400-4857-ad57-2edec5487452" (UID: "e285a82f-a400-4857-ad57-2edec5487452"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:50:34.330741 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.330716    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-model-cache" (OuterVolumeSpecName: "model-cache") pod "e285a82f-a400-4857-ad57-2edec5487452" (UID: "e285a82f-a400-4857-ad57-2edec5487452"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:50:34.330870 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.330852    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:34.330931 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.330875    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:34.330931 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.330888    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:34.332599 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.332569    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e285a82f-a400-4857-ad57-2edec5487452-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e285a82f-a400-4857-ad57-2edec5487452" (UID: "e285a82f-a400-4857-ad57-2edec5487452"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:50:34.332715 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.332604    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-dshm" (OuterVolumeSpecName: "dshm") pod "e285a82f-a400-4857-ad57-2edec5487452" (UID: "e285a82f-a400-4857-ad57-2edec5487452"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:50:34.332715 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.332668    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e285a82f-a400-4857-ad57-2edec5487452-kube-api-access-8pgk6" (OuterVolumeSpecName: "kube-api-access-8pgk6") pod "e285a82f-a400-4857-ad57-2edec5487452" (UID: "e285a82f-a400-4857-ad57-2edec5487452"). InnerVolumeSpecName "kube-api-access-8pgk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:50:34.431959 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.431916    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pgk6\" (UniqueName: \"kubernetes.io/projected/e285a82f-a400-4857-ad57-2edec5487452-kube-api-access-8pgk6\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:34.431959 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.431947    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e285a82f-a400-4857-ad57-2edec5487452-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:34.431959 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.431956    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e285a82f-a400-4857-ad57-2edec5487452-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:50:34.748281 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.748244    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e3392b1-85bd-4001-bd6c-5d757331cc19" path="/var/lib/kubelet/pods/3e3392b1-85bd-4001-bd6c-5d757331cc19/volumes"
Sep 15 21:50:34.785094 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.785057    2575 generic.go:356] "Generic (PLEG): container finished" podID="e285a82f-a400-4857-ad57-2edec5487452" containerID="b47001eb1918d084b2e50830050879fc154f74e9ea5b0ff26116f234cb443c35" exitCode=0
Sep 15 21:50:34.785290 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.785120    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"
Sep 15 21:50:34.785290 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.785141    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k" event={"ID":"e285a82f-a400-4857-ad57-2edec5487452","Type":"ContainerDied","Data":"b47001eb1918d084b2e50830050879fc154f74e9ea5b0ff26116f234cb443c35"}
Sep 15 21:50:34.785290 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.785190    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k" event={"ID":"e285a82f-a400-4857-ad57-2edec5487452","Type":"ContainerDied","Data":"f8e80b571f1eb5812a7c32b9e61c6385a2b3ad0106c1f660172b95a053d26872"}
Sep 15 21:50:34.785290 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.785213    2575 scope.go:117] "RemoveContainer" containerID="b47001eb1918d084b2e50830050879fc154f74e9ea5b0ff26116f234cb443c35"
Sep 15 21:50:34.795097 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.795073    2575 scope.go:117] "RemoveContainer" containerID="b47001eb1918d084b2e50830050879fc154f74e9ea5b0ff26116f234cb443c35"
Sep 15 21:50:34.795483 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:50:34.795461    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47001eb1918d084b2e50830050879fc154f74e9ea5b0ff26116f234cb443c35\": container with ID starting with b47001eb1918d084b2e50830050879fc154f74e9ea5b0ff26116f234cb443c35 not found: ID does not exist" containerID="b47001eb1918d084b2e50830050879fc154f74e9ea5b0ff26116f234cb443c35"
Sep 15 21:50:34.795554 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.795495    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47001eb1918d084b2e50830050879fc154f74e9ea5b0ff26116f234cb443c35"} err="failed to get container status \"b47001eb1918d084b2e50830050879fc154f74e9ea5b0ff26116f234cb443c35\": rpc error: code = NotFound desc = could not find container \"b47001eb1918d084b2e50830050879fc154f74e9ea5b0ff26116f234cb443c35\": container with ID starting with b47001eb1918d084b2e50830050879fc154f74e9ea5b0ff26116f234cb443c35 not found: ID does not exist"
Sep 15 21:50:34.799956 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.799929    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"]
Sep 15 21:50:34.803424 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:34.803393    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/e2e-conflict-a-d70fe03c-kserve-7855fd785-f4v5k"]
Sep 15 21:50:36.748177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:36.748144    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e285a82f-a400-4857-ad57-2edec5487452" path="/var/lib/kubelet/pods/e285a82f-a400-4857-ad57-2edec5487452/volumes"
Sep 15 21:50:56.229070 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:50:56.229025    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:51:01.528547 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.528505    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"]
Sep 15 21:51:01.529008 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.528992    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e285a82f-a400-4857-ad57-2edec5487452" containerName="main"
Sep 15 21:51:01.529008 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529009    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e285a82f-a400-4857-ad57-2edec5487452" containerName="main"
Sep 15 21:51:01.529090 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529021    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c60405-86ab-4edd-8044-7ff6cdcb4416" containerName="revoke-keys"
Sep 15 21:51:01.529090 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529027    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c60405-86ab-4edd-8044-7ff6cdcb4416" containerName="revoke-keys"
Sep 15 21:51:01.529090 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529035    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137c6f2d-c929-4db7-aa12-047afeba8525" containerName="payload-pre-processing"
Sep 15 21:51:01.529090 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529041    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="137c6f2d-c929-4db7-aa12-047afeba8525" containerName="payload-pre-processing"
Sep 15 21:51:01.529090 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529057    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="441c308e-f33d-46c6-8f80-dce21fab23b2" containerName="payload-processing"
Sep 15 21:51:01.529090 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529063    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="441c308e-f33d-46c6-8f80-dce21fab23b2" containerName="payload-processing"
Sep 15 21:51:01.529090 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529074    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b2f3adb-386e-4a67-8859-16bff8ef3928" containerName="main"
Sep 15 21:51:01.529090 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529079    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b2f3adb-386e-4a67-8859-16bff8ef3928" containerName="main"
Sep 15 21:51:01.529090 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529092    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e3392b1-85bd-4001-bd6c-5d757331cc19" containerName="main"
Sep 15 21:51:01.529371 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529097    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e3392b1-85bd-4001-bd6c-5d757331cc19" containerName="main"
Sep 15 21:51:01.529371 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529157    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2c60405-86ab-4edd-8044-7ff6cdcb4416" containerName="revoke-keys"
Sep 15 21:51:01.529371 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529167    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e285a82f-a400-4857-ad57-2edec5487452" containerName="main"
Sep 15 21:51:01.529371 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529175    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="441c308e-f33d-46c6-8f80-dce21fab23b2" containerName="payload-processing"
Sep 15 21:51:01.529371 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529182    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b2f3adb-386e-4a67-8859-16bff8ef3928" containerName="main"
Sep 15 21:51:01.529371 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529188    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e3392b1-85bd-4001-bd6c-5d757331cc19" containerName="main"
Sep 15 21:51:01.529371 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.529194    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="137c6f2d-c929-4db7-aa12-047afeba8525" containerName="payload-pre-processing"
Sep 15 21:51:01.532364 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.532339    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.534686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.534656    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-56e305\"/\"distinct-w4-56e305-kserve-self-signed-certs\""
Sep 15 21:51:01.542942 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.542909    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"]
Sep 15 21:51:01.623365 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.623326    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-dshm\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.623365 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.623369    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-model-cache\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.623596 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.623454    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e26b766-dc93-450d-abf6-676197093a17-tls-certs\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.623596 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.623517    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6wt4\" (UniqueName: \"kubernetes.io/projected/5e26b766-dc93-450d-abf6-676197093a17-kube-api-access-z6wt4\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.623596 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.623567    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-home\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.623699 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.623622    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-tmp-dir\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.724892 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.724843    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-home\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.725104 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.724928    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-tmp-dir\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.725104 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.725002    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-dshm\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.725237 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.725123    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-model-cache\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.725237 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.725225    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e26b766-dc93-450d-abf6-676197093a17-tls-certs\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.725336 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.725264    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wt4\" (UniqueName: \"kubernetes.io/projected/5e26b766-dc93-450d-abf6-676197093a17-kube-api-access-z6wt4\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.725440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.725410    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-tmp-dir\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.725531 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.725500    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-model-cache\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.725669 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.725636    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-home\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.727643 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.727619    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-dshm\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.728076 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.728053    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e26b766-dc93-450d-abf6-676197093a17-tls-certs\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.733344 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.733303    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wt4\" (UniqueName: \"kubernetes.io/projected/5e26b766-dc93-450d-abf6-676197093a17-kube-api-access-z6wt4\") pod \"distinct-w4-56e305-kserve-5854658cd7-478k5\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:01.844227 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:01.844196    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:02.123321 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:02.123285    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"]
Sep 15 21:51:02.125538 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:51:02.125497    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e26b766_dc93_450d_abf6_676197093a17.slice/crio-2de140ce8095833ce750182fd6adafe9219d7e2a9651540776a6373088a908f1 WatchSource:0}: Error finding container 2de140ce8095833ce750182fd6adafe9219d7e2a9651540776a6373088a908f1: Status 404 returned error can't find the container with id 2de140ce8095833ce750182fd6adafe9219d7e2a9651540776a6373088a908f1
Sep 15 21:51:02.955297 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:02.955256    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5" event={"ID":"5e26b766-dc93-450d-abf6-676197093a17","Type":"ContainerStarted","Data":"ca4128fcc61957c22af9d739376b62caab7cdb8af27c99fee984093a3135199c"}
Sep 15 21:51:02.955712 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:02.955308    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5" event={"ID":"5e26b766-dc93-450d-abf6-676197093a17","Type":"ContainerStarted","Data":"2de140ce8095833ce750182fd6adafe9219d7e2a9651540776a6373088a908f1"}
Sep 15 21:51:02.974427 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:02.974364    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5" podStartSLOduration=1.9743437510000001 podStartE2EDuration="1.974343751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:51:02.973501908 +0000 UTC m=+1500.785510355" watchObservedRunningTime="2026-09-15 21:51:02.974343751 +0000 UTC m=+1500.786352193"
Sep 15 21:51:03.700401 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:03.700360    2575 scope.go:117] "RemoveContainer" containerID="ad1282ae4100e8a242c4298007e34ee8e0279ca33143d961089dc440d460135f"
Sep 15 21:51:03.709897 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:03.709873    2575 scope.go:117] "RemoveContainer" containerID="0eb05330b961f4b7e37e98a27e8f5260057df26533faf7f6a41e7509acfa233f"
Sep 15 21:51:03.719741 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:03.719711    2575 scope.go:117] "RemoveContainer" containerID="77d777d4b09f2469538fff8acb8191afbd4921002efc64a30f7fe10641437991"
Sep 15 21:51:11.845389 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:11.845348    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:11.845858 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:11.845407    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:11.858770 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:11.858720    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:12.017444 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:12.017413    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:51:13.736170 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.736117    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"]
Sep 15 21:51:13.741888 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.741855    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.745290 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.745259    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e7a648\"/\"kube-root-ca.crt\""
Sep 15 21:51:13.745484 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.745259    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e7a648\"/\"auth-test-model-e7a648-kserve-self-signed-certs\""
Sep 15 21:51:13.745619 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.745603    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e7a648\"/\"default-dockercfg-vxmvf\""
Sep 15 21:51:13.745693 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.745618    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e7a648\"/\"openshift-service-ca.crt\""
Sep 15 21:51:13.750269 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.750239    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"]
Sep 15 21:51:13.751556 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.751516    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-model-cache\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.751784 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.751611    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def6dd16-fb1a-4d0f-9881-51d147f35833-tls-certs\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.751784 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.751700    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-dshm\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.751784 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.751741    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-home\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.752023 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.751871    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-tmp-dir\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.752023 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.751937    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckq6q\" (UniqueName: \"kubernetes.io/projected/def6dd16-fb1a-4d0f-9881-51d147f35833-kube-api-access-ckq6q\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.852981 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.852944    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def6dd16-fb1a-4d0f-9881-51d147f35833-tls-certs\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.853183 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.852996    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-dshm\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.853247 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.853174    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-home\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.853311 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.853292    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-tmp-dir\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.853392 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.853365    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckq6q\" (UniqueName: \"kubernetes.io/projected/def6dd16-fb1a-4d0f-9881-51d147f35833-kube-api-access-ckq6q\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.853456 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.853443    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-model-cache\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.853610 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.853589    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-home\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.853701 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.853627    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-tmp-dir\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.854032 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.854012    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-model-cache\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.855660 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.855633    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-dshm\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.855968 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.855947    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def6dd16-fb1a-4d0f-9881-51d147f35833-tls-certs\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:13.867887 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:13.867854    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckq6q\" (UniqueName: \"kubernetes.io/projected/def6dd16-fb1a-4d0f-9881-51d147f35833-kube-api-access-ckq6q\") pod \"auth-test-model-e7a648-kserve-d7ffd659-6t4v2\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:14.055356 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:14.055247    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:14.333173 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:14.333141    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"]
Sep 15 21:51:14.335257 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:51:14.335226    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddef6dd16_fb1a_4d0f_9881_51d147f35833.slice/crio-aa901ec2cd443a3f523ff6cfcaea5b08f3c8128f47f525896e8fb805cb497491 WatchSource:0}: Error finding container aa901ec2cd443a3f523ff6cfcaea5b08f3c8128f47f525896e8fb805cb497491: Status 404 returned error can't find the container with id aa901ec2cd443a3f523ff6cfcaea5b08f3c8128f47f525896e8fb805cb497491
Sep 15 21:51:15.028629 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:15.028589    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2" event={"ID":"def6dd16-fb1a-4d0f-9881-51d147f35833","Type":"ContainerStarted","Data":"001e34ed5b24284dc198b821e44a9a28e0ab199fb7edf8cf5cd42e2b59ed525b"}
Sep 15 21:51:15.028629 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:15.028635    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2" event={"ID":"def6dd16-fb1a-4d0f-9881-51d147f35833","Type":"ContainerStarted","Data":"aa901ec2cd443a3f523ff6cfcaea5b08f3c8128f47f525896e8fb805cb497491"}
Sep 15 21:51:15.046251 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:15.046193    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2" podStartSLOduration=2.046173208 podStartE2EDuration="2.046173208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:51:15.044265828 +0000 UTC m=+1512.856274263" watchObservedRunningTime="2026-09-15 21:51:15.046173208 +0000 UTC m=+1512.858181652"
Sep 15 21:51:24.056058 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:24.055945    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:24.056058 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:24.056027    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:24.068808 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:24.068732    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:24.090003 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:24.089967    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:51:30.332890 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:30.332842    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:51:51.764348 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.764310    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"]
Sep 15 21:51:51.768583 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.768548    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.771564 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.771537    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-05557e-openshift-default-dockercfg-wp27k\""
Sep 15 21:51:51.778934 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.778903    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"]
Sep 15 21:51:51.875683 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.875628    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-workload-socket\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.875915 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.875692    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-workload-certs\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.875915 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.875791    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-data\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.875915 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.875882    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-token\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.876030 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.875922    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146136ad-02f1-4df0-aa0c-2ab856b84e83-istiod-ca-cert\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.876030 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.875986    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v742c\" (UniqueName: \"kubernetes.io/projected/146136ad-02f1-4df0-aa0c-2ab856b84e83-kube-api-access-v742c\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.876116 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.876029    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-credential-socket\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.876116 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.876054    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-podinfo\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.876116 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.876092    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-envoy\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.977631 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.977585    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-workload-certs\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.977631 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.977634    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-data\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.977972 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.977709    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-token\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.977972 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.977770    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146136ad-02f1-4df0-aa0c-2ab856b84e83-istiod-ca-cert\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.977972 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.977829    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v742c\" (UniqueName: \"kubernetes.io/projected/146136ad-02f1-4df0-aa0c-2ab856b84e83-kube-api-access-v742c\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.978136 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.977978    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-credential-socket\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.978136 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.978030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-podinfo\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.978136 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.978066    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-workload-certs\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.978136 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.978075    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-envoy\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.978136 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.978106    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-data\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.978402 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.978167    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-workload-socket\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.978455 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.978398    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-credential-socket\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.978492 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.978454    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-workload-socket\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.978561 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.978538    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146136ad-02f1-4df0-aa0c-2ab856b84e83-istiod-ca-cert\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.980553 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.980527    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-envoy\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.980832 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.980814    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-podinfo\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.985441 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.985412    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-token\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:51.985592 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:51.985559    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v742c\" (UniqueName: \"kubernetes.io/projected/146136ad-02f1-4df0-aa0c-2ab856b84e83-kube-api-access-v742c\") pod \"e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") " pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:52.083175 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:52.083069    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:52.365135 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:52.365105    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"]
Sep 15 21:51:52.366980 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:51:52.366948    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146136ad_02f1_4df0_aa0c_2ab856b84e83.slice/crio-014f6c29203d836b9675a76cde4128cc97799163ee45d5b8eafb6bedf87e2efe WatchSource:0}: Error finding container 014f6c29203d836b9675a76cde4128cc97799163ee45d5b8eafb6bedf87e2efe: Status 404 returned error can't find the container with id 014f6c29203d836b9675a76cde4128cc97799163ee45d5b8eafb6bedf87e2efe
Sep 15 21:51:52.369514 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:52.369475    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:51:52.369630 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:52.369552    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:51:52.369630 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:52.369606    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:51:53.246311 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:53.246257    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b" event={"ID":"146136ad-02f1-4df0-aa0c-2ab856b84e83","Type":"ContainerStarted","Data":"ba61199f0f5e516123d87a18cfb01c3255ae2104257c71db5ff187137ebb4546"}
Sep 15 21:51:53.246736 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:53.246333    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b" event={"ID":"146136ad-02f1-4df0-aa0c-2ab856b84e83","Type":"ContainerStarted","Data":"014f6c29203d836b9675a76cde4128cc97799163ee45d5b8eafb6bedf87e2efe"}
Sep 15 21:51:53.264846 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:53.264782    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b" podStartSLOduration=2.26474505 podStartE2EDuration="2.26474505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:51:53.263619422 +0000 UTC m=+1551.075627908" watchObservedRunningTime="2026-09-15 21:51:53.26474505 +0000 UTC m=+1551.076753574"
Sep 15 21:51:54.083667 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:54.083620    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:54.090101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:54.090068    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:54.252084 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:54.252048    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:51:54.253146 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:51:54.253127    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:52:00.417081 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.417043    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"]
Sep 15 21:52:00.421157 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.421132    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"
Sep 15 21:52:00.423346 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.423305    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-05557e\""
Sep 15 21:52:00.423512 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.423369    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-05557e-dockercfg-2s7l2\""
Sep 15 21:52:00.431283 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.431250    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"]
Sep 15 21:52:00.457190 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.457139    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"]
Sep 15 21:52:00.461066 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.461036    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"
Sep 15 21:52:00.470205 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.470173    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"]
Sep 15 21:52:00.575370 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.575335    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a02b4e3-4c53-46ae-ab24-942fdee6ed98-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk\" (UID: \"5a02b4e3-4c53-46ae-ab24-942fdee6ed98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"
Sep 15 21:52:00.575573 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.575401    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcafe98d-dc66-4f02-88d4-a324df03bb2a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-05557e-7756dbc97-snsgb\" (UID: \"bcafe98d-dc66-4f02-88d4-a324df03bb2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"
Sep 15 21:52:00.575573 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.575541    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5kkv\" (UniqueName: \"kubernetes.io/projected/bcafe98d-dc66-4f02-88d4-a324df03bb2a-kube-api-access-g5kkv\") pod \"payload-processing-e2e-api-a-05557e-7756dbc97-snsgb\" (UID: \"bcafe98d-dc66-4f02-88d4-a324df03bb2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"
Sep 15 21:52:00.575647 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.575583    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5nrp\" (UniqueName: \"kubernetes.io/projected/5a02b4e3-4c53-46ae-ab24-942fdee6ed98-kube-api-access-g5nrp\") pod \"payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk\" (UID: \"5a02b4e3-4c53-46ae-ab24-942fdee6ed98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"
Sep 15 21:52:00.676450 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.676353    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5kkv\" (UniqueName: \"kubernetes.io/projected/bcafe98d-dc66-4f02-88d4-a324df03bb2a-kube-api-access-g5kkv\") pod \"payload-processing-e2e-api-a-05557e-7756dbc97-snsgb\" (UID: \"bcafe98d-dc66-4f02-88d4-a324df03bb2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"
Sep 15 21:52:00.676450 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.676407    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5nrp\" (UniqueName: \"kubernetes.io/projected/5a02b4e3-4c53-46ae-ab24-942fdee6ed98-kube-api-access-g5nrp\") pod \"payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk\" (UID: \"5a02b4e3-4c53-46ae-ab24-942fdee6ed98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"
Sep 15 21:52:00.676662 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.676468    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a02b4e3-4c53-46ae-ab24-942fdee6ed98-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk\" (UID: \"5a02b4e3-4c53-46ae-ab24-942fdee6ed98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"
Sep 15 21:52:00.676662 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.676541    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcafe98d-dc66-4f02-88d4-a324df03bb2a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-05557e-7756dbc97-snsgb\" (UID: \"bcafe98d-dc66-4f02-88d4-a324df03bb2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"
Sep 15 21:52:00.677230 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.677210    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a02b4e3-4c53-46ae-ab24-942fdee6ed98-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk\" (UID: \"5a02b4e3-4c53-46ae-ab24-942fdee6ed98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"
Sep 15 21:52:00.677322 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.677305    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcafe98d-dc66-4f02-88d4-a324df03bb2a-plugins-config-volume\") pod \"payload-processing-e2e-api-a-05557e-7756dbc97-snsgb\" (UID: \"bcafe98d-dc66-4f02-88d4-a324df03bb2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"
Sep 15 21:52:00.684144 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.684109    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5nrp\" (UniqueName: \"kubernetes.io/projected/5a02b4e3-4c53-46ae-ab24-942fdee6ed98-kube-api-access-g5nrp\") pod \"payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk\" (UID: \"5a02b4e3-4c53-46ae-ab24-942fdee6ed98\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"
Sep 15 21:52:00.684144 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.684110    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5kkv\" (UniqueName: \"kubernetes.io/projected/bcafe98d-dc66-4f02-88d4-a324df03bb2a-kube-api-access-g5kkv\") pod \"payload-processing-e2e-api-a-05557e-7756dbc97-snsgb\" (UID: \"bcafe98d-dc66-4f02-88d4-a324df03bb2a\") " pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"
Sep 15 21:52:00.735242 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.735199    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"
Sep 15 21:52:00.774080 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:00.774041    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"
Sep 15 21:52:01.017026 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:01.016994    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"]
Sep 15 21:52:01.018904 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:52:01.018873    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a02b4e3_4c53_46ae_ab24_942fdee6ed98.slice/crio-0044956c1d620d5186eb7dd88f7fc8a8953c8af0aca57c836b31460e8611557f WatchSource:0}: Error finding container 0044956c1d620d5186eb7dd88f7fc8a8953c8af0aca57c836b31460e8611557f: Status 404 returned error can't find the container with id 0044956c1d620d5186eb7dd88f7fc8a8953c8af0aca57c836b31460e8611557f
Sep 15 21:52:01.062976 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:01.062950    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"]
Sep 15 21:52:01.065210 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:52:01.065187    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcafe98d_dc66_4f02_88d4_a324df03bb2a.slice/crio-927de7bda27f999269713c63f8cc512fd547a5a300d0a173ee534e63e18729e4 WatchSource:0}: Error finding container 927de7bda27f999269713c63f8cc512fd547a5a300d0a173ee534e63e18729e4: Status 404 returned error can't find the container with id 927de7bda27f999269713c63f8cc512fd547a5a300d0a173ee534e63e18729e4
Sep 15 21:52:01.299257 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:01.299145    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk" event={"ID":"5a02b4e3-4c53-46ae-ab24-942fdee6ed98","Type":"ContainerStarted","Data":"b9b32539498948b021683c576dba99bc6fd6c1b8da32759414e726835115f1a3"}
Sep 15 21:52:01.299257 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:01.299205    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk" event={"ID":"5a02b4e3-4c53-46ae-ab24-942fdee6ed98","Type":"ContainerStarted","Data":"0044956c1d620d5186eb7dd88f7fc8a8953c8af0aca57c836b31460e8611557f"}
Sep 15 21:52:01.299501 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:01.299255    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"
Sep 15 21:52:01.301340 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:01.301305    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb" event={"ID":"bcafe98d-dc66-4f02-88d4-a324df03bb2a","Type":"ContainerStarted","Data":"7555da9efaf63ba66f60d95b662a7a722ab63ce72a7e1d33c16c3d1779429a1e"}
Sep 15 21:52:01.301531 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:01.301352    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb" event={"ID":"bcafe98d-dc66-4f02-88d4-a324df03bb2a","Type":"ContainerStarted","Data":"927de7bda27f999269713c63f8cc512fd547a5a300d0a173ee534e63e18729e4"}
Sep 15 21:52:01.301531 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:01.301370    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"
Sep 15 21:52:01.314429 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:01.314355    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk" podStartSLOduration=1.314329282 podStartE2EDuration="1.314329282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:52:01.312658186 +0000 UTC m=+1559.124666634" watchObservedRunningTime="2026-09-15 21:52:01.314329282 +0000 UTC m=+1559.126337727"
Sep 15 21:52:01.329086 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:01.329014    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb" podStartSLOduration=1.328991041 podStartE2EDuration="1.328991041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:52:01.327067984 +0000 UTC m=+1559.139076429" watchObservedRunningTime="2026-09-15 21:52:01.328991041 +0000 UTC m=+1559.140999486"
Sep 15 21:52:02.315380 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.315337    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"]
Sep 15 21:52:02.319106 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.319084    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.321187 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.321161    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-56e305\"/\"distinct-2-w4-56e305-kserve-self-signed-certs\""
Sep 15 21:52:02.327440 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.327403    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"]
Sep 15 21:52:02.396464 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.396420    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-home\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.396667 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.396477    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-tls-certs\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.396667 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.396527    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zd9j\" (UniqueName: \"kubernetes.io/projected/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-kube-api-access-9zd9j\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.396667 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.396561    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-tmp-dir\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.396895 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.396866    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-model-cache\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.396997 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.396984    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-dshm\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.498067 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.498024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-model-cache\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.498272 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.498114    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-dshm\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.498272 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.498191    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-home\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.498272 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.498218    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-tls-certs\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.498272 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.498253    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zd9j\" (UniqueName: \"kubernetes.io/projected/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-kube-api-access-9zd9j\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.498516 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.498285    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-tmp-dir\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.498516 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.498492    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-model-cache\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.498613 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.498571    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-home\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.498654 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.498642    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-tmp-dir\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.500730 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.500702    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-dshm\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.501039 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.501019    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-tls-certs\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.506369 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.506338    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zd9j\" (UniqueName: \"kubernetes.io/projected/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-kube-api-access-9zd9j\") pod \"distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") " pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.631670 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.631633    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:02.910343 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:02.910311    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"]
Sep 15 21:52:02.912497 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:52:02.912459    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0a857bb_1788_4de0_b3cf_0bebbb5b1243.slice/crio-bade718401b54c67b8da774855fa41eb16488728fee6ef46c8baedccb8ac23f1 WatchSource:0}: Error finding container bade718401b54c67b8da774855fa41eb16488728fee6ef46c8baedccb8ac23f1: Status 404 returned error can't find the container with id bade718401b54c67b8da774855fa41eb16488728fee6ef46c8baedccb8ac23f1
Sep 15 21:52:03.315542 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:03.315442    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m" event={"ID":"a0a857bb-1788-4de0-b3cf-0bebbb5b1243","Type":"ContainerStarted","Data":"1bdbaf275ae912b0c75cf2c349ab350e6eb5efb965241d951ffde9052e7c2578"}
Sep 15 21:52:03.315542 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:03.315493    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m" event={"ID":"a0a857bb-1788-4de0-b3cf-0bebbb5b1243","Type":"ContainerStarted","Data":"bade718401b54c67b8da774855fa41eb16488728fee6ef46c8baedccb8ac23f1"}
Sep 15 21:52:03.332190 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:03.332134    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m" podStartSLOduration=1.3321153510000001 podStartE2EDuration="1.332115351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:52:03.329726939 +0000 UTC m=+1561.141735384" watchObservedRunningTime="2026-09-15 21:52:03.332115351 +0000 UTC m=+1561.144123794"
Sep 15 21:52:06.131059 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:06.131011    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:52:12.309989 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:12.309952    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"
Sep 15 21:52:12.310391 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:12.310045    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"
Sep 15 21:52:12.631764 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:12.631718    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:12.631950 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:12.631827    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:12.645273 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:12.645237    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:13.394390 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:13.394358    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:52:37.530392 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:37.530355    2575 generic.go:356] "Generic (PLEG): container finished" podID="146136ad-02f1-4df0-aa0c-2ab856b84e83" containerID="ba61199f0f5e516123d87a18cfb01c3255ae2104257c71db5ff187137ebb4546" exitCode=0
Sep 15 21:52:37.530959 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:37.530436    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b" event={"ID":"146136ad-02f1-4df0-aa0c-2ab856b84e83","Type":"ContainerDied","Data":"ba61199f0f5e516123d87a18cfb01c3255ae2104257c71db5ff187137ebb4546"}
Sep 15 21:52:37.531095 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:37.531075    2575 scope.go:117] "RemoveContainer" containerID="ba61199f0f5e516123d87a18cfb01c3255ae2104257c71db5ff187137ebb4546"
Sep 15 21:52:37.533454 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:37.533397    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:52:37.533643 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:37.533585    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:52:37.533742 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:37.533694    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:52:38.539844 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:38.539804    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b" event={"ID":"146136ad-02f1-4df0-aa0c-2ab856b84e83","Type":"ContainerStarted","Data":"af380fa60524b3a307563e65b6548cac9de14d1d1ac9dab14ca5cb4633b8c0d3"}
Sep 15 21:52:39.084291 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:39.084244    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:52:39.085417 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:39.085389    2575 patch_prober.go:28] interesting pod/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 15 21:52:39.085555 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:39.085460    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b" podUID="146136ad-02f1-4df0-aa0c-2ab856b84e83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 15 21:52:39.252570 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:39.252514    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:52:40.083674 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:40.083635    2575 patch_prober.go:28] interesting pod/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 15 21:52:40.084211 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:40.083703    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b" podUID="146136ad-02f1-4df0-aa0c-2ab856b84e83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 15 21:52:41.084391 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:41.084354    2575 patch_prober.go:28] interesting pod/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 15 21:52:41.084847 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:41.084411    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b" podUID="146136ad-02f1-4df0-aa0c-2ab856b84e83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 15 21:52:42.084478 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:42.084436    2575 patch_prober.go:28] interesting pod/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 15 21:52:42.084891 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:42.084499    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b" podUID="146136ad-02f1-4df0-aa0c-2ab856b84e83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 15 21:52:43.083870 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.083831    2575 patch_prober.go:28] interesting pod/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body=
Sep 15 21:52:43.084075 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.083894    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b" podUID="146136ad-02f1-4df0-aa0c-2ab856b84e83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused"
Sep 15 21:52:43.415445 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.415402    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"]
Sep 15 21:52:43.419326 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.419304    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.422403 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.422369    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-360e43\"/\"default-dockercfg-nz88v\""
Sep 15 21:52:43.422403 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.422390    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-360e43\"/\"embedding-w0-360e43-kserve-self-signed-certs\""
Sep 15 21:52:43.422629 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.422397    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-360e43\"/\"openshift-service-ca.crt\""
Sep 15 21:52:43.422629 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.422382    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-360e43\"/\"kube-root-ca.crt\""
Sep 15 21:52:43.426973 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.426944    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"]
Sep 15 21:52:43.444400 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.444362    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htjbf\" (UniqueName: \"kubernetes.io/projected/31162923-010d-46fc-8522-f51eb79ab9f4-kube-api-access-htjbf\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.444633 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.444455    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-dshm\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.444633 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.444477    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-tmp-dir\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.444633 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.444554    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-home\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.444633 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.444605    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-model-cache\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.444916 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.444637    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31162923-010d-46fc-8522-f51eb79ab9f4-tls-certs\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.545245 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.545202    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-home\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.545451 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.545268    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-model-cache\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.545451 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.545294    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31162923-010d-46fc-8522-f51eb79ab9f4-tls-certs\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.545451 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.545325    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htjbf\" (UniqueName: \"kubernetes.io/projected/31162923-010d-46fc-8522-f51eb79ab9f4-kube-api-access-htjbf\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.545630 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.545470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-dshm\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.545630 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.545500    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-tmp-dir\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.545775 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.545703    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-home\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.545775 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.545703    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-model-cache\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.545865 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.545855    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-tmp-dir\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.547930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.547905    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-dshm\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.548314 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.548289    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31162923-010d-46fc-8522-f51eb79ab9f4-tls-certs\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.554266 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.554193    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htjbf\" (UniqueName: \"kubernetes.io/projected/31162923-010d-46fc-8522-f51eb79ab9f4-kube-api-access-htjbf\") pod \"embedding-w0-360e43-kserve-75fbbd46f7-jzbn4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") " pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:43.730504 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:43.730399    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:44.032619 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:44.032583    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"]
Sep 15 21:52:44.034434 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:52:44.034395    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31162923_010d_46fc_8522_f51eb79ab9f4.slice/crio-1fdcee0232ba402acdd000d12a4e1939a8ecee97e992552e77ab7c449bf3c60f WatchSource:0}: Error finding container 1fdcee0232ba402acdd000d12a4e1939a8ecee97e992552e77ab7c449bf3c60f: Status 404 returned error can't find the container with id 1fdcee0232ba402acdd000d12a4e1939a8ecee97e992552e77ab7c449bf3c60f
Sep 15 21:52:44.088561 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:44.088529    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:52:44.089889 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:44.089863    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:52:44.589278 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:44.589237    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4" event={"ID":"31162923-010d-46fc-8522-f51eb79ab9f4","Type":"ContainerStarted","Data":"bedeb4e4b664a4cabc984460c7131d8248bc64937b9fe28b3ef4b54973ecbcf1"}
Sep 15 21:52:44.589278 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:44.589282    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4" event={"ID":"31162923-010d-46fc-8522-f51eb79ab9f4","Type":"ContainerStarted","Data":"1fdcee0232ba402acdd000d12a4e1939a8ecee97e992552e77ab7c449bf3c60f"}
Sep 15 21:52:44.606878 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:44.606813    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4" podStartSLOduration=1.6067914779999999 podStartE2EDuration="1.606791478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:52:44.60401568 +0000 UTC m=+1602.416024117" watchObservedRunningTime="2026-09-15 21:52:44.606791478 +0000 UTC m=+1602.418799957"
Sep 15 21:52:45.231184 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:45.231135    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:52:53.731226 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:53.731125    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:53.731226 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:53.731203    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:53.744350 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:53.744318    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:52:54.663307 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:52:54.663269    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:53:02.491515 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:02.491468    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l"]
Sep 15 21:53:02.495733 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:02.495701    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l"
Sep 15 21:53:02.498991 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:02.498953    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 21:53:02.499378 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:02.499354    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:53:02.502732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:02.502697    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l"]
Sep 15 21:53:02.562748 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:02.562709    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fee5a2d-3c2b-4a88-99f3-899b9115c3a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-05557e-cg25l\" (UID: \"6fee5a2d-3c2b-4a88-99f3-899b9115c3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l"
Sep 15 21:53:02.663899 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:02.663861    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fee5a2d-3c2b-4a88-99f3-899b9115c3a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-05557e-cg25l\" (UID: \"6fee5a2d-3c2b-4a88-99f3-899b9115c3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l"
Sep 15 21:53:02.666251 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:02.666228    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:53:02.675652 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:02.675589    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fee5a2d-3c2b-4a88-99f3-899b9115c3a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-05557e-cg25l\" (UID: \"6fee5a2d-3c2b-4a88-99f3-899b9115c3a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l"
Sep 15 21:53:02.814964 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:02.814880    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 21:53:02.823801 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:02.823740    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l"
Sep 15 21:53:03.099827 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:03.099799    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l"]
Sep 15 21:53:03.101939 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:53:03.101906    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fee5a2d_3c2b_4a88_99f3_899b9115c3a3.slice/crio-66f5d4c8336072ce0d021f8c71533c4345fd45fe166985b3f41100aa14608bcb WatchSource:0}: Error finding container 66f5d4c8336072ce0d021f8c71533c4345fd45fe166985b3f41100aa14608bcb: Status 404 returned error can't find the container with id 66f5d4c8336072ce0d021f8c71533c4345fd45fe166985b3f41100aa14608bcb
Sep 15 21:53:03.711033 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:03.710932    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l" event={"ID":"6fee5a2d-3c2b-4a88-99f3-899b9115c3a3","Type":"ContainerStarted","Data":"5063639d297927563a8c89749015c14d672d40b9e6ebe9398c672f23f75b0e9c"}
Sep 15 21:53:03.711033 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:03.711031    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l" event={"ID":"6fee5a2d-3c2b-4a88-99f3-899b9115c3a3","Type":"ContainerStarted","Data":"66f5d4c8336072ce0d021f8c71533c4345fd45fe166985b3f41100aa14608bcb"}
Sep 15 21:53:03.727431 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:03.727373    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l" podStartSLOduration=1.727350046 podStartE2EDuration="1.727350046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:53:03.725207136 +0000 UTC m=+1621.537215581" watchObservedRunningTime="2026-09-15 21:53:03.727350046 +0000 UTC m=+1621.539358493"
Sep 15 21:53:04.719370 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:04.719333    2575 generic.go:356] "Generic (PLEG): container finished" podID="6fee5a2d-3c2b-4a88-99f3-899b9115c3a3" containerID="5063639d297927563a8c89749015c14d672d40b9e6ebe9398c672f23f75b0e9c" exitCode=0
Sep 15 21:53:04.719837 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:04.719416    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l" event={"ID":"6fee5a2d-3c2b-4a88-99f3-899b9115c3a3","Type":"ContainerDied","Data":"5063639d297927563a8c89749015c14d672d40b9e6ebe9398c672f23f75b0e9c"}
Sep 15 21:53:05.887121 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:05.887084    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l"
Sep 15 21:53:06.003682 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:06.003575    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fee5a2d-3c2b-4a88-99f3-899b9115c3a3-maas-api-service-ca\") pod \"6fee5a2d-3c2b-4a88-99f3-899b9115c3a3\" (UID: \"6fee5a2d-3c2b-4a88-99f3-899b9115c3a3\") "
Sep 15 21:53:06.004074 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:06.004039    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fee5a2d-3c2b-4a88-99f3-899b9115c3a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6fee5a2d-3c2b-4a88-99f3-899b9115c3a3" (UID: "6fee5a2d-3c2b-4a88-99f3-899b9115c3a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:53:06.105494 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:06.105443    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fee5a2d-3c2b-4a88-99f3-899b9115c3a3-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:06.733380 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:06.733347    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l" event={"ID":"6fee5a2d-3c2b-4a88-99f3-899b9115c3a3","Type":"ContainerDied","Data":"66f5d4c8336072ce0d021f8c71533c4345fd45fe166985b3f41100aa14608bcb"}
Sep 15 21:53:06.733566 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:06.733401    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66f5d4c8336072ce0d021f8c71533c4345fd45fe166985b3f41100aa14608bcb"
Sep 15 21:53:06.733566 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:06.733359    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l"
Sep 15 21:53:18.397977 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.397935    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"]
Sep 15 21:53:18.398453 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.398205    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb" podUID="bcafe98d-dc66-4f02-88d4-a324df03bb2a" containerName="payload-processing" containerID="cri-o://7555da9efaf63ba66f60d95b662a7a722ab63ce72a7e1d33c16c3d1779429a1e" gracePeriod=30
Sep 15 21:53:18.409442 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.409401    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"]
Sep 15 21:53:18.410787 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.410727    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk" podUID="5a02b4e3-4c53-46ae-ab24-942fdee6ed98" containerName="payload-pre-processing" containerID="cri-o://b9b32539498948b021683c576dba99bc6fd6c1b8da32759414e726835115f1a3" gracePeriod=30
Sep 15 21:53:18.503482 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:53:18.503441    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a02b4e3_4c53_46ae_ab24_942fdee6ed98.slice/crio-conmon-b9b32539498948b021683c576dba99bc6fd6c1b8da32759414e726835115f1a3.scope\": RecentStats: unable to find data in memory cache]"
Sep 15 21:53:18.551582 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.551539    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l"]
Sep 15 21:53:18.555429 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.555385    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-05557e-cg25l"]
Sep 15 21:53:18.735886 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.735859    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"
Sep 15 21:53:18.739583 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.739554    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"
Sep 15 21:53:18.748489 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.748455    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fee5a2d-3c2b-4a88-99f3-899b9115c3a3" path="/var/lib/kubelet/pods/6fee5a2d-3c2b-4a88-99f3-899b9115c3a3/volumes"
Sep 15 21:53:18.825420 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.825378    2575 generic.go:356] "Generic (PLEG): container finished" podID="5a02b4e3-4c53-46ae-ab24-942fdee6ed98" containerID="b9b32539498948b021683c576dba99bc6fd6c1b8da32759414e726835115f1a3" exitCode=0
Sep 15 21:53:18.825641 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.825455    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"
Sep 15 21:53:18.825641 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.825480    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk" event={"ID":"5a02b4e3-4c53-46ae-ab24-942fdee6ed98","Type":"ContainerDied","Data":"b9b32539498948b021683c576dba99bc6fd6c1b8da32759414e726835115f1a3"}
Sep 15 21:53:18.825641 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.825586    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk" event={"ID":"5a02b4e3-4c53-46ae-ab24-942fdee6ed98","Type":"ContainerDied","Data":"0044956c1d620d5186eb7dd88f7fc8a8953c8af0aca57c836b31460e8611557f"}
Sep 15 21:53:18.825641 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.825622    2575 scope.go:117] "RemoveContainer" containerID="b9b32539498948b021683c576dba99bc6fd6c1b8da32759414e726835115f1a3"
Sep 15 21:53:18.827371 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.827345    2575 generic.go:356] "Generic (PLEG): container finished" podID="bcafe98d-dc66-4f02-88d4-a324df03bb2a" containerID="7555da9efaf63ba66f60d95b662a7a722ab63ce72a7e1d33c16c3d1779429a1e" exitCode=0
Sep 15 21:53:18.827470 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.827419    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb" event={"ID":"bcafe98d-dc66-4f02-88d4-a324df03bb2a","Type":"ContainerDied","Data":"7555da9efaf63ba66f60d95b662a7a722ab63ce72a7e1d33c16c3d1779429a1e"}
Sep 15 21:53:18.827517 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.827469    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb" event={"ID":"bcafe98d-dc66-4f02-88d4-a324df03bb2a","Type":"ContainerDied","Data":"927de7bda27f999269713c63f8cc512fd547a5a300d0a173ee534e63e18729e4"}
Sep 15 21:53:18.827517 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.827447    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"
Sep 15 21:53:18.838020 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.837990    2575 scope.go:117] "RemoveContainer" containerID="b9b32539498948b021683c576dba99bc6fd6c1b8da32759414e726835115f1a3"
Sep 15 21:53:18.838410 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:53:18.838391    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9b32539498948b021683c576dba99bc6fd6c1b8da32759414e726835115f1a3\": container with ID starting with b9b32539498948b021683c576dba99bc6fd6c1b8da32759414e726835115f1a3 not found: ID does not exist" containerID="b9b32539498948b021683c576dba99bc6fd6c1b8da32759414e726835115f1a3"
Sep 15 21:53:18.838467 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.838420    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9b32539498948b021683c576dba99bc6fd6c1b8da32759414e726835115f1a3"} err="failed to get container status \"b9b32539498948b021683c576dba99bc6fd6c1b8da32759414e726835115f1a3\": rpc error: code = NotFound desc = could not find container \"b9b32539498948b021683c576dba99bc6fd6c1b8da32759414e726835115f1a3\": container with ID starting with b9b32539498948b021683c576dba99bc6fd6c1b8da32759414e726835115f1a3 not found: ID does not exist"
Sep 15 21:53:18.838467 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.838438    2575 scope.go:117] "RemoveContainer" containerID="7555da9efaf63ba66f60d95b662a7a722ab63ce72a7e1d33c16c3d1779429a1e"
Sep 15 21:53:18.848874 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.848842    2575 scope.go:117] "RemoveContainer" containerID="7555da9efaf63ba66f60d95b662a7a722ab63ce72a7e1d33c16c3d1779429a1e"
Sep 15 21:53:18.849066 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.849046    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5nrp\" (UniqueName: \"kubernetes.io/projected/5a02b4e3-4c53-46ae-ab24-942fdee6ed98-kube-api-access-g5nrp\") pod \"5a02b4e3-4c53-46ae-ab24-942fdee6ed98\" (UID: \"5a02b4e3-4c53-46ae-ab24-942fdee6ed98\") "
Sep 15 21:53:18.849120 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.849081    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcafe98d-dc66-4f02-88d4-a324df03bb2a-plugins-config-volume\") pod \"bcafe98d-dc66-4f02-88d4-a324df03bb2a\" (UID: \"bcafe98d-dc66-4f02-88d4-a324df03bb2a\") "
Sep 15 21:53:18.849207 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.849190    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5kkv\" (UniqueName: \"kubernetes.io/projected/bcafe98d-dc66-4f02-88d4-a324df03bb2a-kube-api-access-g5kkv\") pod \"bcafe98d-dc66-4f02-88d4-a324df03bb2a\" (UID: \"bcafe98d-dc66-4f02-88d4-a324df03bb2a\") "
Sep 15 21:53:18.849313 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:53:18.849289    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7555da9efaf63ba66f60d95b662a7a722ab63ce72a7e1d33c16c3d1779429a1e\": container with ID starting with 7555da9efaf63ba66f60d95b662a7a722ab63ce72a7e1d33c16c3d1779429a1e not found: ID does not exist" containerID="7555da9efaf63ba66f60d95b662a7a722ab63ce72a7e1d33c16c3d1779429a1e"
Sep 15 21:53:18.849395 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.849319    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a02b4e3-4c53-46ae-ab24-942fdee6ed98-plugins-config-volume\") pod \"5a02b4e3-4c53-46ae-ab24-942fdee6ed98\" (UID: \"5a02b4e3-4c53-46ae-ab24-942fdee6ed98\") "
Sep 15 21:53:18.849395 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.849326    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7555da9efaf63ba66f60d95b662a7a722ab63ce72a7e1d33c16c3d1779429a1e"} err="failed to get container status \"7555da9efaf63ba66f60d95b662a7a722ab63ce72a7e1d33c16c3d1779429a1e\": rpc error: code = NotFound desc = could not find container \"7555da9efaf63ba66f60d95b662a7a722ab63ce72a7e1d33c16c3d1779429a1e\": container with ID starting with 7555da9efaf63ba66f60d95b662a7a722ab63ce72a7e1d33c16c3d1779429a1e not found: ID does not exist"
Sep 15 21:53:18.849641 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.849613    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcafe98d-dc66-4f02-88d4-a324df03bb2a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bcafe98d-dc66-4f02-88d4-a324df03bb2a" (UID: "bcafe98d-dc66-4f02-88d4-a324df03bb2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:53:18.849747 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.849664    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a02b4e3-4c53-46ae-ab24-942fdee6ed98-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5a02b4e3-4c53-46ae-ab24-942fdee6ed98" (UID: "5a02b4e3-4c53-46ae-ab24-942fdee6ed98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:53:18.850401 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.850375    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a02b4e3-4c53-46ae-ab24-942fdee6ed98-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:18.850509 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.850407    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcafe98d-dc66-4f02-88d4-a324df03bb2a-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:18.851743 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.851719    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a02b4e3-4c53-46ae-ab24-942fdee6ed98-kube-api-access-g5nrp" (OuterVolumeSpecName: "kube-api-access-g5nrp") pod "5a02b4e3-4c53-46ae-ab24-942fdee6ed98" (UID: "5a02b4e3-4c53-46ae-ab24-942fdee6ed98"). InnerVolumeSpecName "kube-api-access-g5nrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:53:18.852152 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.852132    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcafe98d-dc66-4f02-88d4-a324df03bb2a-kube-api-access-g5kkv" (OuterVolumeSpecName: "kube-api-access-g5kkv") pod "bcafe98d-dc66-4f02-88d4-a324df03bb2a" (UID: "bcafe98d-dc66-4f02-88d4-a324df03bb2a"). InnerVolumeSpecName "kube-api-access-g5kkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:53:18.951686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.951637    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5nrp\" (UniqueName: \"kubernetes.io/projected/5a02b4e3-4c53-46ae-ab24-942fdee6ed98-kube-api-access-g5nrp\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:18.951686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:18.951681    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5kkv\" (UniqueName: \"kubernetes.io/projected/bcafe98d-dc66-4f02-88d4-a324df03bb2a-kube-api-access-g5kkv\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:19.148278 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:19.148222    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"]
Sep 15 21:53:19.151456 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:19.151420    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-05557e-58c65c7f86-8f8fk"]
Sep 15 21:53:19.163634 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:19.163588    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"]
Sep 15 21:53:19.166020 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:19.165992    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-05557e-7756dbc97-snsgb"]
Sep 15 21:53:20.748854 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:20.748814    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a02b4e3-4c53-46ae-ab24-942fdee6ed98" path="/var/lib/kubelet/pods/5a02b4e3-4c53-46ae-ab24-942fdee6ed98/volumes"
Sep 15 21:53:20.749240 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:20.749213    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcafe98d-dc66-4f02-88d4-a324df03bb2a" path="/var/lib/kubelet/pods/bcafe98d-dc66-4f02-88d4-a324df03bb2a/volumes"
Sep 15 21:53:25.183581 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.183542    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"]
Sep 15 21:53:25.184018 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.183844    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b" podUID="146136ad-02f1-4df0-aa0c-2ab856b84e83" containerName="istio-proxy" containerID="cri-o://af380fa60524b3a307563e65b6548cac9de14d1d1ac9dab14ca5cb4633b8c0d3" gracePeriod=30
Sep 15 21:53:25.568944 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.568838    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l"]
Sep 15 21:53:25.569482 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.569459    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fee5a2d-3c2b-4a88-99f3-899b9115c3a3" containerName="revoke-keys"
Sep 15 21:53:25.569530 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.569485    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fee5a2d-3c2b-4a88-99f3-899b9115c3a3" containerName="revoke-keys"
Sep 15 21:53:25.569530 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.569502    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a02b4e3-4c53-46ae-ab24-942fdee6ed98" containerName="payload-pre-processing"
Sep 15 21:53:25.569530 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.569508    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a02b4e3-4c53-46ae-ab24-942fdee6ed98" containerName="payload-pre-processing"
Sep 15 21:53:25.569530 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.569514    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcafe98d-dc66-4f02-88d4-a324df03bb2a" containerName="payload-processing"
Sep 15 21:53:25.569530 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.569521    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcafe98d-dc66-4f02-88d4-a324df03bb2a" containerName="payload-processing"
Sep 15 21:53:25.569699 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.569603    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a02b4e3-4c53-46ae-ab24-942fdee6ed98" containerName="payload-pre-processing"
Sep 15 21:53:25.569699 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.569610    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcafe98d-dc66-4f02-88d4-a324df03bb2a" containerName="payload-processing"
Sep 15 21:53:25.569699 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.569619    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fee5a2d-3c2b-4a88-99f3-899b9115c3a3" containerName="revoke-keys"
Sep 15 21:53:25.574473 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.574445    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l"
Sep 15 21:53:25.577232 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.577067    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:53:25.577232 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.577093    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 21:53:25.579908 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.579871    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l"]
Sep 15 21:53:25.628082 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.628030    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219fcbf3-b352-4e80-91dc-f54740589180-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-74da27-zwd2l\" (UID: \"219fcbf3-b352-4e80-91dc-f54740589180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l"
Sep 15 21:53:25.728996 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.728951    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219fcbf3-b352-4e80-91dc-f54740589180-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-74da27-zwd2l\" (UID: \"219fcbf3-b352-4e80-91dc-f54740589180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l"
Sep 15 21:53:25.729612 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.729586    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219fcbf3-b352-4e80-91dc-f54740589180-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-74da27-zwd2l\" (UID: \"219fcbf3-b352-4e80-91dc-f54740589180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l"
Sep 15 21:53:25.890882 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:25.890843    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l"
Sep 15 21:53:26.171714 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:26.171683    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l"]
Sep 15 21:53:26.173521 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:53:26.173481    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219fcbf3_b352_4e80_91dc_f54740589180.slice/crio-bc7a0589efd5bde86aa2635d665e24f8c676ce31a6041bd9aced2c18dbca01d2 WatchSource:0}: Error finding container bc7a0589efd5bde86aa2635d665e24f8c676ce31a6041bd9aced2c18dbca01d2: Status 404 returned error can't find the container with id bc7a0589efd5bde86aa2635d665e24f8c676ce31a6041bd9aced2c18dbca01d2
Sep 15 21:53:26.882178 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:26.882143    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l" event={"ID":"219fcbf3-b352-4e80-91dc-f54740589180","Type":"ContainerStarted","Data":"c163f9d031c94b6fddf452aa30529674d3527f7a9efd0ccc8caceb48920ac8ea"}
Sep 15 21:53:26.882566 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:26.882203    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l" event={"ID":"219fcbf3-b352-4e80-91dc-f54740589180","Type":"ContainerStarted","Data":"bc7a0589efd5bde86aa2635d665e24f8c676ce31a6041bd9aced2c18dbca01d2"}
Sep 15 21:53:26.899333 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:26.899273    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l" podStartSLOduration=1.8992532130000002 podStartE2EDuration="1.899253213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:53:26.897414979 +0000 UTC m=+1644.709423423" watchObservedRunningTime="2026-09-15 21:53:26.899253213 +0000 UTC m=+1644.711261702"
Sep 15 21:53:27.894197 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:27.894158    2575 generic.go:356] "Generic (PLEG): container finished" podID="219fcbf3-b352-4e80-91dc-f54740589180" containerID="c163f9d031c94b6fddf452aa30529674d3527f7a9efd0ccc8caceb48920ac8ea" exitCode=0
Sep 15 21:53:27.894601 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:27.894251    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l" event={"ID":"219fcbf3-b352-4e80-91dc-f54740589180","Type":"ContainerDied","Data":"c163f9d031c94b6fddf452aa30529674d3527f7a9efd0ccc8caceb48920ac8ea"}
Sep 15 21:53:29.050297 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:29.050260    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l"
Sep 15 21:53:29.091280 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:29.091236    2575 patch_prober.go:28] interesting pod/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:53:29.091435 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:29.091320    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b" podUID="146136ad-02f1-4df0-aa0c-2ab856b84e83" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:53:29.167829 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:29.167731    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219fcbf3-b352-4e80-91dc-f54740589180-maas-api-service-ca\") pod \"219fcbf3-b352-4e80-91dc-f54740589180\" (UID: \"219fcbf3-b352-4e80-91dc-f54740589180\") "
Sep 15 21:53:29.168126 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:29.168099    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219fcbf3-b352-4e80-91dc-f54740589180-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "219fcbf3-b352-4e80-91dc-f54740589180" (UID: "219fcbf3-b352-4e80-91dc-f54740589180"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:53:29.269774 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:29.269721    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219fcbf3-b352-4e80-91dc-f54740589180-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:29.907168 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:29.907134    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l"
Sep 15 21:53:29.907547 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:29.907173    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l" event={"ID":"219fcbf3-b352-4e80-91dc-f54740589180","Type":"ContainerDied","Data":"bc7a0589efd5bde86aa2635d665e24f8c676ce31a6041bd9aced2c18dbca01d2"}
Sep 15 21:53:29.907547 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:29.907223    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc7a0589efd5bde86aa2635d665e24f8c676ce31a6041bd9aced2c18dbca01d2"
Sep 15 21:53:30.455914 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.455830    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:53:30.584890 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.584848    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-workload-socket\") pod \"146136ad-02f1-4df0-aa0c-2ab856b84e83\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") "
Sep 15 21:53:30.584890 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.584891    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-data\") pod \"146136ad-02f1-4df0-aa0c-2ab856b84e83\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") "
Sep 15 21:53:30.585148 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.584979    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-workload-certs\") pod \"146136ad-02f1-4df0-aa0c-2ab856b84e83\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") "
Sep 15 21:53:30.585148 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.585009    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-token\") pod \"146136ad-02f1-4df0-aa0c-2ab856b84e83\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") "
Sep 15 21:53:30.585148 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.585043    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-envoy\") pod \"146136ad-02f1-4df0-aa0c-2ab856b84e83\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") "
Sep 15 21:53:30.585148 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.585062    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146136ad-02f1-4df0-aa0c-2ab856b84e83-istiod-ca-cert\") pod \"146136ad-02f1-4df0-aa0c-2ab856b84e83\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") "
Sep 15 21:53:30.585148 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.585095    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v742c\" (UniqueName: \"kubernetes.io/projected/146136ad-02f1-4df0-aa0c-2ab856b84e83-kube-api-access-v742c\") pod \"146136ad-02f1-4df0-aa0c-2ab856b84e83\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") "
Sep 15 21:53:30.585148 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.585121    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-credential-socket\") pod \"146136ad-02f1-4df0-aa0c-2ab856b84e83\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") "
Sep 15 21:53:30.585450 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.585154    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-podinfo\") pod \"146136ad-02f1-4df0-aa0c-2ab856b84e83\" (UID: \"146136ad-02f1-4df0-aa0c-2ab856b84e83\") "
Sep 15 21:53:30.585450 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.585216    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "146136ad-02f1-4df0-aa0c-2ab856b84e83" (UID: "146136ad-02f1-4df0-aa0c-2ab856b84e83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:53:30.585450 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.585230    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-data" (OuterVolumeSpecName: "istio-data") pod "146136ad-02f1-4df0-aa0c-2ab856b84e83" (UID: "146136ad-02f1-4df0-aa0c-2ab856b84e83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:53:30.585450 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.585425    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "146136ad-02f1-4df0-aa0c-2ab856b84e83" (UID: "146136ad-02f1-4df0-aa0c-2ab856b84e83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:53:30.585657 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.585511    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "146136ad-02f1-4df0-aa0c-2ab856b84e83" (UID: "146136ad-02f1-4df0-aa0c-2ab856b84e83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:53:30.585657 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.585620    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:30.585805 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.585660    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:30.585805 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.585677    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:30.585805 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.585701    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:30.585805 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.585617    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146136ad-02f1-4df0-aa0c-2ab856b84e83-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "146136ad-02f1-4df0-aa0c-2ab856b84e83" (UID: "146136ad-02f1-4df0-aa0c-2ab856b84e83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:53:30.587501 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.587473    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146136ad-02f1-4df0-aa0c-2ab856b84e83-kube-api-access-v742c" (OuterVolumeSpecName: "kube-api-access-v742c") pod "146136ad-02f1-4df0-aa0c-2ab856b84e83" (UID: "146136ad-02f1-4df0-aa0c-2ab856b84e83"). InnerVolumeSpecName "kube-api-access-v742c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:53:30.587501 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.587484    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-token" (OuterVolumeSpecName: "istio-token") pod "146136ad-02f1-4df0-aa0c-2ab856b84e83" (UID: "146136ad-02f1-4df0-aa0c-2ab856b84e83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:53:30.587935 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.587917    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "146136ad-02f1-4df0-aa0c-2ab856b84e83" (UID: "146136ad-02f1-4df0-aa0c-2ab856b84e83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:53:30.588008 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.587970    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "146136ad-02f1-4df0-aa0c-2ab856b84e83" (UID: "146136ad-02f1-4df0-aa0c-2ab856b84e83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:53:30.687014 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.686970    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:30.687014 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.687004    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:30.687014 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.687013    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146136ad-02f1-4df0-aa0c-2ab856b84e83-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:30.687014 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.687021    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v742c\" (UniqueName: \"kubernetes.io/projected/146136ad-02f1-4df0-aa0c-2ab856b84e83-kube-api-access-v742c\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:30.687014 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.687030    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146136ad-02f1-4df0-aa0c-2ab856b84e83-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:30.915584 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.915541    2575 generic.go:356] "Generic (PLEG): container finished" podID="146136ad-02f1-4df0-aa0c-2ab856b84e83" containerID="af380fa60524b3a307563e65b6548cac9de14d1d1ac9dab14ca5cb4633b8c0d3" exitCode=0
Sep 15 21:53:30.915802 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.915603    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"
Sep 15 21:53:30.915802 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.915597    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b" event={"ID":"146136ad-02f1-4df0-aa0c-2ab856b84e83","Type":"ContainerDied","Data":"af380fa60524b3a307563e65b6548cac9de14d1d1ac9dab14ca5cb4633b8c0d3"}
Sep 15 21:53:30.915802 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.915740    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b" event={"ID":"146136ad-02f1-4df0-aa0c-2ab856b84e83","Type":"ContainerDied","Data":"014f6c29203d836b9675a76cde4128cc97799163ee45d5b8eafb6bedf87e2efe"}
Sep 15 21:53:30.915802 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.915780    2575 scope.go:117] "RemoveContainer" containerID="af380fa60524b3a307563e65b6548cac9de14d1d1ac9dab14ca5cb4633b8c0d3"
Sep 15 21:53:30.926509 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.926483    2575 scope.go:117] "RemoveContainer" containerID="ba61199f0f5e516123d87a18cfb01c3255ae2104257c71db5ff187137ebb4546"
Sep 15 21:53:30.933828 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.933797    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"]
Sep 15 21:53:30.937407 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.937368    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-05557e-openshift-default-7487957cf8-w9v7b"]
Sep 15 21:53:30.939906 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.939881    2575 scope.go:117] "RemoveContainer" containerID="af380fa60524b3a307563e65b6548cac9de14d1d1ac9dab14ca5cb4633b8c0d3"
Sep 15 21:53:30.940244 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:53:30.940226    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af380fa60524b3a307563e65b6548cac9de14d1d1ac9dab14ca5cb4633b8c0d3\": container with ID starting with af380fa60524b3a307563e65b6548cac9de14d1d1ac9dab14ca5cb4633b8c0d3 not found: ID does not exist" containerID="af380fa60524b3a307563e65b6548cac9de14d1d1ac9dab14ca5cb4633b8c0d3"
Sep 15 21:53:30.940300 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.940255    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af380fa60524b3a307563e65b6548cac9de14d1d1ac9dab14ca5cb4633b8c0d3"} err="failed to get container status \"af380fa60524b3a307563e65b6548cac9de14d1d1ac9dab14ca5cb4633b8c0d3\": rpc error: code = NotFound desc = could not find container \"af380fa60524b3a307563e65b6548cac9de14d1d1ac9dab14ca5cb4633b8c0d3\": container with ID starting with af380fa60524b3a307563e65b6548cac9de14d1d1ac9dab14ca5cb4633b8c0d3 not found: ID does not exist"
Sep 15 21:53:30.940300 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.940274    2575 scope.go:117] "RemoveContainer" containerID="ba61199f0f5e516123d87a18cfb01c3255ae2104257c71db5ff187137ebb4546"
Sep 15 21:53:30.940569 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:53:30.940553    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba61199f0f5e516123d87a18cfb01c3255ae2104257c71db5ff187137ebb4546\": container with ID starting with ba61199f0f5e516123d87a18cfb01c3255ae2104257c71db5ff187137ebb4546 not found: ID does not exist" containerID="ba61199f0f5e516123d87a18cfb01c3255ae2104257c71db5ff187137ebb4546"
Sep 15 21:53:30.940631 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:30.940571    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba61199f0f5e516123d87a18cfb01c3255ae2104257c71db5ff187137ebb4546"} err="failed to get container status \"ba61199f0f5e516123d87a18cfb01c3255ae2104257c71db5ff187137ebb4546\": rpc error: code = NotFound desc = could not find container \"ba61199f0f5e516123d87a18cfb01c3255ae2104257c71db5ff187137ebb4546\": container with ID starting with ba61199f0f5e516123d87a18cfb01c3255ae2104257c71db5ff187137ebb4546 not found: ID does not exist"
Sep 15 21:53:32.749409 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:32.749371    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="146136ad-02f1-4df0-aa0c-2ab856b84e83" path="/var/lib/kubelet/pods/146136ad-02f1-4df0-aa0c-2ab856b84e83/volumes"
Sep 15 21:53:41.179611 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:41.179561    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l"]
Sep 15 21:53:41.182025 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:41.181994    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-74da27-zwd2l"]
Sep 15 21:53:42.748644 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:42.748601    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="219fcbf3-b352-4e80-91dc-f54740589180" path="/var/lib/kubelet/pods/219fcbf3-b352-4e80-91dc-f54740589180/volumes"
Sep 15 21:53:48.122771 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.122722    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd"]
Sep 15 21:53:48.123239 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.123223    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146136ad-02f1-4df0-aa0c-2ab856b84e83" containerName="istio-proxy"
Sep 15 21:53:48.123239 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.123240    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="146136ad-02f1-4df0-aa0c-2ab856b84e83" containerName="istio-proxy"
Sep 15 21:53:48.123322 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.123262    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146136ad-02f1-4df0-aa0c-2ab856b84e83" containerName="istio-proxy"
Sep 15 21:53:48.123322 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.123268    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="146136ad-02f1-4df0-aa0c-2ab856b84e83" containerName="istio-proxy"
Sep 15 21:53:48.123322 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.123282    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="219fcbf3-b352-4e80-91dc-f54740589180" containerName="revoke-keys"
Sep 15 21:53:48.123322 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.123288    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="219fcbf3-b352-4e80-91dc-f54740589180" containerName="revoke-keys"
Sep 15 21:53:48.123448 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.123360    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="146136ad-02f1-4df0-aa0c-2ab856b84e83" containerName="istio-proxy"
Sep 15 21:53:48.123448 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.123369    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="219fcbf3-b352-4e80-91dc-f54740589180" containerName="revoke-keys"
Sep 15 21:53:48.123448 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.123375    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="146136ad-02f1-4df0-aa0c-2ab856b84e83" containerName="istio-proxy"
Sep 15 21:53:48.127913 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.127888    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd"
Sep 15 21:53:48.130202 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.130166    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:53:48.130382 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.130248    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 21:53:48.132523 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.132478    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd"]
Sep 15 21:53:48.178313 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.178272    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd\" (UID: \"faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd"
Sep 15 21:53:48.279596 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.279547    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd\" (UID: \"faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd"
Sep 15 21:53:48.280164 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.280144    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd\" (UID: \"faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd"
Sep 15 21:53:48.442573 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.442475    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd"
Sep 15 21:53:48.717166 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:48.717137    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd"]
Sep 15 21:53:48.720345 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:53:48.720312    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa348dc_8fb4_473b_9ba1_d2d5b95fb1ed.slice/crio-7a8e0163c674265c144369e7d993b75e5c30d97bfcbecbcd1477d2e38a6a229f WatchSource:0}: Error finding container 7a8e0163c674265c144369e7d993b75e5c30d97bfcbecbcd1477d2e38a6a229f: Status 404 returned error can't find the container with id 7a8e0163c674265c144369e7d993b75e5c30d97bfcbecbcd1477d2e38a6a229f
Sep 15 21:53:49.031491 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:49.031398    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd" event={"ID":"faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed","Type":"ContainerStarted","Data":"7a8e0163c674265c144369e7d993b75e5c30d97bfcbecbcd1477d2e38a6a229f"}
Sep 15 21:53:50.039546 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:50.039507    2575 generic.go:356] "Generic (PLEG): container finished" podID="faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed" containerID="5b1ab5720f3489a8250ba5646e3ace08592005817e7fb47a48fb556161867013" exitCode=0
Sep 15 21:53:50.039992 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:50.039590    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd" event={"ID":"faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed","Type":"ContainerDied","Data":"5b1ab5720f3489a8250ba5646e3ace08592005817e7fb47a48fb556161867013"}
Sep 15 21:53:51.185586 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:51.185554    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd"
Sep 15 21:53:51.315671 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:51.315568    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed-maas-api-service-ca\") pod \"faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed\" (UID: \"faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed\") "
Sep 15 21:53:51.316060 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:51.316029    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed" (UID: "faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:53:51.417343 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:51.417300    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:52.054050 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:52.054007    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd" event={"ID":"faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed","Type":"ContainerDied","Data":"7a8e0163c674265c144369e7d993b75e5c30d97bfcbecbcd1477d2e38a6a229f"}
Sep 15 21:53:52.054050 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:52.054048    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd"
Sep 15 21:53:52.054276 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:52.054071    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a8e0163c674265c144369e7d993b75e5c30d97bfcbecbcd1477d2e38a6a229f"
Sep 15 21:53:57.929315 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:57.929266    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:53:58.580041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:58.580000    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"]
Sep 15 21:53:58.580596 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:58.580522    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg" podUID="f0ad6e4f-bfad-48fb-943c-183c31673c1b" containerName="payload-processing" containerID="cri-o://e0b0be6b4f9b7c379370d67a658b2b8101106419fb5633efca038f28e25a7c78" gracePeriod=30
Sep 15 21:53:58.587855 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:58.587672    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"]
Sep 15 21:53:58.588060 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:58.587948    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb" podUID="87fd9284-2fa6-4d55-83e1-6fe45b4957c5" containerName="payload-pre-processing" containerID="cri-o://286ba67327208d1000b5426b06772d819578a57284da184113cb1feff9efcff5" gracePeriod=30
Sep 15 21:53:58.726817 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:58.726498    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd"]
Sep 15 21:53:58.729043 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:58.729004    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-850f22-lfhzd"]
Sep 15 21:53:58.752348 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:58.752024    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed" path="/var/lib/kubelet/pods/faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed/volumes"
Sep 15 21:53:58.896287 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:58.896256    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"
Sep 15 21:53:58.899140 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:58.899113    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"
Sep 15 21:53:59.007695 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.007653    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ad6e4f-bfad-48fb-943c-183c31673c1b-plugins-config-volume\") pod \"f0ad6e4f-bfad-48fb-943c-183c31673c1b\" (UID: \"f0ad6e4f-bfad-48fb-943c-183c31673c1b\") "
Sep 15 21:53:59.008242 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.007707    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztskb\" (UniqueName: \"kubernetes.io/projected/87fd9284-2fa6-4d55-83e1-6fe45b4957c5-kube-api-access-ztskb\") pod \"87fd9284-2fa6-4d55-83e1-6fe45b4957c5\" (UID: \"87fd9284-2fa6-4d55-83e1-6fe45b4957c5\") "
Sep 15 21:53:59.008242 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.007890    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5sm4\" (UniqueName: \"kubernetes.io/projected/f0ad6e4f-bfad-48fb-943c-183c31673c1b-kube-api-access-r5sm4\") pod \"f0ad6e4f-bfad-48fb-943c-183c31673c1b\" (UID: \"f0ad6e4f-bfad-48fb-943c-183c31673c1b\") "
Sep 15 21:53:59.008242 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.007940    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87fd9284-2fa6-4d55-83e1-6fe45b4957c5-plugins-config-volume\") pod \"87fd9284-2fa6-4d55-83e1-6fe45b4957c5\" (UID: \"87fd9284-2fa6-4d55-83e1-6fe45b4957c5\") "
Sep 15 21:53:59.008429 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.008280    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87fd9284-2fa6-4d55-83e1-6fe45b4957c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "87fd9284-2fa6-4d55-83e1-6fe45b4957c5" (UID: "87fd9284-2fa6-4d55-83e1-6fe45b4957c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:53:59.008429 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.008291    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ad6e4f-bfad-48fb-943c-183c31673c1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f0ad6e4f-bfad-48fb-943c-183c31673c1b" (UID: "f0ad6e4f-bfad-48fb-943c-183c31673c1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:53:59.010845 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.010809    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ad6e4f-bfad-48fb-943c-183c31673c1b-kube-api-access-r5sm4" (OuterVolumeSpecName: "kube-api-access-r5sm4") pod "f0ad6e4f-bfad-48fb-943c-183c31673c1b" (UID: "f0ad6e4f-bfad-48fb-943c-183c31673c1b"). InnerVolumeSpecName "kube-api-access-r5sm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:53:59.011014 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.010928    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87fd9284-2fa6-4d55-83e1-6fe45b4957c5-kube-api-access-ztskb" (OuterVolumeSpecName: "kube-api-access-ztskb") pod "87fd9284-2fa6-4d55-83e1-6fe45b4957c5" (UID: "87fd9284-2fa6-4d55-83e1-6fe45b4957c5"). InnerVolumeSpecName "kube-api-access-ztskb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:53:59.098046 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.098011    2575 generic.go:356] "Generic (PLEG): container finished" podID="f0ad6e4f-bfad-48fb-943c-183c31673c1b" containerID="e0b0be6b4f9b7c379370d67a658b2b8101106419fb5633efca038f28e25a7c78" exitCode=0
Sep 15 21:53:59.098214 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.098061    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"
Sep 15 21:53:59.098214 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.098105    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg" event={"ID":"f0ad6e4f-bfad-48fb-943c-183c31673c1b","Type":"ContainerDied","Data":"e0b0be6b4f9b7c379370d67a658b2b8101106419fb5633efca038f28e25a7c78"}
Sep 15 21:53:59.098214 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.098165    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg" event={"ID":"f0ad6e4f-bfad-48fb-943c-183c31673c1b","Type":"ContainerDied","Data":"7b7a90e6923b4f9769cb27c7eef9504e7f7b95d86746c96a0805e8df28710ac8"}
Sep 15 21:53:59.098214 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.098199    2575 scope.go:117] "RemoveContainer" containerID="e0b0be6b4f9b7c379370d67a658b2b8101106419fb5633efca038f28e25a7c78"
Sep 15 21:53:59.099732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.099712    2575 generic.go:356] "Generic (PLEG): container finished" podID="87fd9284-2fa6-4d55-83e1-6fe45b4957c5" containerID="286ba67327208d1000b5426b06772d819578a57284da184113cb1feff9efcff5" exitCode=0
Sep 15 21:53:59.099858 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.099784    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"
Sep 15 21:53:59.099858 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.099786    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb" event={"ID":"87fd9284-2fa6-4d55-83e1-6fe45b4957c5","Type":"ContainerDied","Data":"286ba67327208d1000b5426b06772d819578a57284da184113cb1feff9efcff5"}
Sep 15 21:53:59.099858 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.099839    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb" event={"ID":"87fd9284-2fa6-4d55-83e1-6fe45b4957c5","Type":"ContainerDied","Data":"0ec7049cc4d1e7486b0dbbf209e0fbc69f3b2116db1f7515077e027f74aa3740"}
Sep 15 21:53:59.108723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.108698    2575 scope.go:117] "RemoveContainer" containerID="e0b0be6b4f9b7c379370d67a658b2b8101106419fb5633efca038f28e25a7c78"
Sep 15 21:53:59.109195 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:53:59.109167    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b0be6b4f9b7c379370d67a658b2b8101106419fb5633efca038f28e25a7c78\": container with ID starting with e0b0be6b4f9b7c379370d67a658b2b8101106419fb5633efca038f28e25a7c78 not found: ID does not exist" containerID="e0b0be6b4f9b7c379370d67a658b2b8101106419fb5633efca038f28e25a7c78"
Sep 15 21:53:59.109296 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.109202    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b0be6b4f9b7c379370d67a658b2b8101106419fb5633efca038f28e25a7c78"} err="failed to get container status \"e0b0be6b4f9b7c379370d67a658b2b8101106419fb5633efca038f28e25a7c78\": rpc error: code = NotFound desc = could not find container \"e0b0be6b4f9b7c379370d67a658b2b8101106419fb5633efca038f28e25a7c78\": container with ID starting with e0b0be6b4f9b7c379370d67a658b2b8101106419fb5633efca038f28e25a7c78 not found: ID does not exist"
Sep 15 21:53:59.109296 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.109222    2575 scope.go:117] "RemoveContainer" containerID="286ba67327208d1000b5426b06772d819578a57284da184113cb1feff9efcff5"
Sep 15 21:53:59.109429 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.109304    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ad6e4f-bfad-48fb-943c-183c31673c1b-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:59.109429 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.109330    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztskb\" (UniqueName: \"kubernetes.io/projected/87fd9284-2fa6-4d55-83e1-6fe45b4957c5-kube-api-access-ztskb\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:59.109429 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.109345    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5sm4\" (UniqueName: \"kubernetes.io/projected/f0ad6e4f-bfad-48fb-943c-183c31673c1b-kube-api-access-r5sm4\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:59.109429 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.109360    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87fd9284-2fa6-4d55-83e1-6fe45b4957c5-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:53:59.119195 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.119173    2575 scope.go:117] "RemoveContainer" containerID="286ba67327208d1000b5426b06772d819578a57284da184113cb1feff9efcff5"
Sep 15 21:53:59.119612 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:53:59.119591    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286ba67327208d1000b5426b06772d819578a57284da184113cb1feff9efcff5\": container with ID starting with 286ba67327208d1000b5426b06772d819578a57284da184113cb1feff9efcff5 not found: ID does not exist" containerID="286ba67327208d1000b5426b06772d819578a57284da184113cb1feff9efcff5"
Sep 15 21:53:59.119663 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.119622    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286ba67327208d1000b5426b06772d819578a57284da184113cb1feff9efcff5"} err="failed to get container status \"286ba67327208d1000b5426b06772d819578a57284da184113cb1feff9efcff5\": rpc error: code = NotFound desc = could not find container \"286ba67327208d1000b5426b06772d819578a57284da184113cb1feff9efcff5\": container with ID starting with 286ba67327208d1000b5426b06772d819578a57284da184113cb1feff9efcff5 not found: ID does not exist"
Sep 15 21:53:59.123848 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.123812    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"]
Sep 15 21:53:59.127707 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.127674    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-850f22-8686d6f775-f75gg"]
Sep 15 21:53:59.136336 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.136298    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"]
Sep 15 21:53:59.142131 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:53:59.142097    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-850f22-cc956d8b-z4nmb"]
Sep 15 21:54:00.748090 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:00.748051    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87fd9284-2fa6-4d55-83e1-6fe45b4957c5" path="/var/lib/kubelet/pods/87fd9284-2fa6-4d55-83e1-6fe45b4957c5/volumes"
Sep 15 21:54:00.748686 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:00.748662    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0ad6e4f-bfad-48fb-943c-183c31673c1b" path="/var/lib/kubelet/pods/f0ad6e4f-bfad-48fb-943c-183c31673c1b/volumes"
Sep 15 21:54:05.244123 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.244081    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"]
Sep 15 21:54:05.244630 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.244323    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q" podUID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" containerName="istio-proxy" containerID="cri-o://9932d3be156c2352ed062afd480d660db95c49bece7e2cc20ae7ae4a286c1838" gracePeriod=30
Sep 15 21:54:05.608862 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.608798    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2"]
Sep 15 21:54:05.609390 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.609368    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed" containerName="revoke-keys"
Sep 15 21:54:05.609482 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.609393    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed" containerName="revoke-keys"
Sep 15 21:54:05.609482 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.609421    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87fd9284-2fa6-4d55-83e1-6fe45b4957c5" containerName="payload-pre-processing"
Sep 15 21:54:05.609482 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.609430    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="87fd9284-2fa6-4d55-83e1-6fe45b4957c5" containerName="payload-pre-processing"
Sep 15 21:54:05.609482 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.609450    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ad6e4f-bfad-48fb-943c-183c31673c1b" containerName="payload-processing"
Sep 15 21:54:05.609482 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.609459    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ad6e4f-bfad-48fb-943c-183c31673c1b" containerName="payload-processing"
Sep 15 21:54:05.609748 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.609561    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ad6e4f-bfad-48fb-943c-183c31673c1b" containerName="payload-processing"
Sep 15 21:54:05.609748 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.609576    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="87fd9284-2fa6-4d55-83e1-6fe45b4957c5" containerName="payload-pre-processing"
Sep 15 21:54:05.609748 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.609591    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="faa348dc-8fb4-473b-9ba1-d2d5b95fb1ed" containerName="revoke-keys"
Sep 15 21:54:05.614404 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.614368    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2"
Sep 15 21:54:05.616814 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.616774    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:54:05.616981 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.616827    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 21:54:05.619022 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.618991    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2"]
Sep 15 21:54:05.785149 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.785109    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ed83c0-588b-4f54-9dc4-027f8639621f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2\" (UID: \"26ed83c0-588b-4f54-9dc4-027f8639621f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2"
Sep 15 21:54:05.886746 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.886623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ed83c0-588b-4f54-9dc4-027f8639621f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2\" (UID: \"26ed83c0-588b-4f54-9dc4-027f8639621f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2"
Sep 15 21:54:05.887339 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.887317    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ed83c0-588b-4f54-9dc4-027f8639621f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2\" (UID: \"26ed83c0-588b-4f54-9dc4-027f8639621f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2"
Sep 15 21:54:05.930013 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:05.929964    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2"
Sep 15 21:54:06.205739 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:06.205703    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2"]
Sep 15 21:54:06.208666 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:54:06.208632    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ed83c0_588b_4f54_9dc4_027f8639621f.slice/crio-b408c796d76be4e6bf55b7bb6eb1adc6a86d4e68b08e7eb4e0e8bfa8d06b3230 WatchSource:0}: Error finding container b408c796d76be4e6bf55b7bb6eb1adc6a86d4e68b08e7eb4e0e8bfa8d06b3230: Status 404 returned error can't find the container with id b408c796d76be4e6bf55b7bb6eb1adc6a86d4e68b08e7eb4e0e8bfa8d06b3230
Sep 15 21:54:06.730152 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:06.730116    2575 patch_prober.go:28] interesting pod/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 21:54:06.730587 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:06.730180    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q" podUID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 21:54:07.153120 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:07.153083    2575 generic.go:356] "Generic (PLEG): container finished" podID="26ed83c0-588b-4f54-9dc4-027f8639621f" containerID="90eaac38faa7150e16477061b36baec66953080c5ef0fdecee4315332a61314b" exitCode=0
Sep 15 21:54:07.153320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:07.153169    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2" event={"ID":"26ed83c0-588b-4f54-9dc4-027f8639621f","Type":"ContainerDied","Data":"90eaac38faa7150e16477061b36baec66953080c5ef0fdecee4315332a61314b"}
Sep 15 21:54:07.153320 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:07.153226    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2" event={"ID":"26ed83c0-588b-4f54-9dc4-027f8639621f","Type":"ContainerStarted","Data":"b408c796d76be4e6bf55b7bb6eb1adc6a86d4e68b08e7eb4e0e8bfa8d06b3230"}
Sep 15 21:54:08.309968 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:08.309926    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2"
Sep 15 21:54:08.416017 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:08.415912    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ed83c0-588b-4f54-9dc4-027f8639621f-maas-api-service-ca\") pod \"26ed83c0-588b-4f54-9dc4-027f8639621f\" (UID: \"26ed83c0-588b-4f54-9dc4-027f8639621f\") "
Sep 15 21:54:08.416322 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:08.416299    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26ed83c0-588b-4f54-9dc4-027f8639621f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "26ed83c0-588b-4f54-9dc4-027f8639621f" (UID: "26ed83c0-588b-4f54-9dc4-027f8639621f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:54:08.517860 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:08.517814    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/26ed83c0-588b-4f54-9dc4-027f8639621f-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:09.166865 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:09.166829    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2"
Sep 15 21:54:09.166865 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:09.166864    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2" event={"ID":"26ed83c0-588b-4f54-9dc4-027f8639621f","Type":"ContainerDied","Data":"b408c796d76be4e6bf55b7bb6eb1adc6a86d4e68b08e7eb4e0e8bfa8d06b3230"}
Sep 15 21:54:09.167104 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:09.166910    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b408c796d76be4e6bf55b7bb6eb1adc6a86d4e68b08e7eb4e0e8bfa8d06b3230"
Sep 15 21:54:10.516483 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.516398    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:54:10.538276 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.538242    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-workload-certs\") pod \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") "
Sep 15 21:54:10.538485 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.538298    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-data\") pod \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") "
Sep 15 21:54:10.538485 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.538357    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vft7\" (UniqueName: \"kubernetes.io/projected/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-kube-api-access-9vft7\") pod \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") "
Sep 15 21:54:10.538648 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.538595    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f48b61d5-9f9c-4527-b08c-c3fd2523bc98" (UID: "f48b61d5-9f9c-4527-b08c-c3fd2523bc98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:54:10.538824 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.538801    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-podinfo\") pod \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") "
Sep 15 21:54:10.538898 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.538847    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-data" (OuterVolumeSpecName: "istio-data") pod "f48b61d5-9f9c-4527-b08c-c3fd2523bc98" (UID: "f48b61d5-9f9c-4527-b08c-c3fd2523bc98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:54:10.538968 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.538930    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istiod-ca-cert\") pod \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") "
Sep 15 21:54:10.538968 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.538962    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-envoy\") pod \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") "
Sep 15 21:54:10.539082 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.539020    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-credential-socket\") pod \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") "
Sep 15 21:54:10.539082 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.539075    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-token\") pod \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") "
Sep 15 21:54:10.539193 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.539100    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-workload-socket\") pod \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\" (UID: \"f48b61d5-9f9c-4527-b08c-c3fd2523bc98\") "
Sep 15 21:54:10.539519 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.539456    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f48b61d5-9f9c-4527-b08c-c3fd2523bc98" (UID: "f48b61d5-9f9c-4527-b08c-c3fd2523bc98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:54:10.539519 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.539480    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:10.539519 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.539497    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:10.539807 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.539786    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f48b61d5-9f9c-4527-b08c-c3fd2523bc98" (UID: "f48b61d5-9f9c-4527-b08c-c3fd2523bc98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:54:10.539875 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.539836    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f48b61d5-9f9c-4527-b08c-c3fd2523bc98" (UID: "f48b61d5-9f9c-4527-b08c-c3fd2523bc98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:54:10.542438 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.542401    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f48b61d5-9f9c-4527-b08c-c3fd2523bc98" (UID: "f48b61d5-9f9c-4527-b08c-c3fd2523bc98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 21:54:10.543732 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.543704    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f48b61d5-9f9c-4527-b08c-c3fd2523bc98" (UID: "f48b61d5-9f9c-4527-b08c-c3fd2523bc98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:54:10.543879 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.543748    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-token" (OuterVolumeSpecName: "istio-token") pod "f48b61d5-9f9c-4527-b08c-c3fd2523bc98" (UID: "f48b61d5-9f9c-4527-b08c-c3fd2523bc98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:54:10.543879 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.543771    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-kube-api-access-9vft7" (OuterVolumeSpecName: "kube-api-access-9vft7") pod "f48b61d5-9f9c-4527-b08c-c3fd2523bc98" (UID: "f48b61d5-9f9c-4527-b08c-c3fd2523bc98"). InnerVolumeSpecName "kube-api-access-9vft7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:54:10.640155 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.640117    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:10.640155 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.640148    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:10.640155 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.640158    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:10.640155 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.640166    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:10.640441 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.640175    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:10.640441 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.640184    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vft7\" (UniqueName: \"kubernetes.io/projected/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-kube-api-access-9vft7\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:10.640441 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:10.640192    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f48b61d5-9f9c-4527-b08c-c3fd2523bc98-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:11.181115 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:11.181079    2575 generic.go:356] "Generic (PLEG): container finished" podID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" containerID="9932d3be156c2352ed062afd480d660db95c49bece7e2cc20ae7ae4a286c1838" exitCode=0
Sep 15 21:54:11.181338 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:11.181132    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"
Sep 15 21:54:11.181338 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:11.181158    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q" event={"ID":"f48b61d5-9f9c-4527-b08c-c3fd2523bc98","Type":"ContainerDied","Data":"9932d3be156c2352ed062afd480d660db95c49bece7e2cc20ae7ae4a286c1838"}
Sep 15 21:54:11.181338 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:11.181222    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q" event={"ID":"f48b61d5-9f9c-4527-b08c-c3fd2523bc98","Type":"ContainerDied","Data":"c703cd519e92ad902183b857cd577c218a7c84d6195d400fdf357bc6335b1af1"}
Sep 15 21:54:11.181338 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:11.181253    2575 scope.go:117] "RemoveContainer" containerID="9932d3be156c2352ed062afd480d660db95c49bece7e2cc20ae7ae4a286c1838"
Sep 15 21:54:11.191893 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:11.191872    2575 scope.go:117] "RemoveContainer" containerID="631bc24a6282ac61000fabbaa87b0e99fe9e10f69154cb35a2da53ee1d10c320"
Sep 15 21:54:11.197924 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:11.197885    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"]
Sep 15 21:54:11.203255 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:11.203230    2575 scope.go:117] "RemoveContainer" containerID="9932d3be156c2352ed062afd480d660db95c49bece7e2cc20ae7ae4a286c1838"
Sep 15 21:54:11.203657 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:54:11.203634    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9932d3be156c2352ed062afd480d660db95c49bece7e2cc20ae7ae4a286c1838\": container with ID starting with 9932d3be156c2352ed062afd480d660db95c49bece7e2cc20ae7ae4a286c1838 not found: ID does not exist" containerID="9932d3be156c2352ed062afd480d660db95c49bece7e2cc20ae7ae4a286c1838"
Sep 15 21:54:11.203776 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:11.203670    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9932d3be156c2352ed062afd480d660db95c49bece7e2cc20ae7ae4a286c1838"} err="failed to get container status \"9932d3be156c2352ed062afd480d660db95c49bece7e2cc20ae7ae4a286c1838\": rpc error: code = NotFound desc = could not find container \"9932d3be156c2352ed062afd480d660db95c49bece7e2cc20ae7ae4a286c1838\": container with ID starting with 9932d3be156c2352ed062afd480d660db95c49bece7e2cc20ae7ae4a286c1838 not found: ID does not exist"
Sep 15 21:54:11.203776 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:11.203693    2575 scope.go:117] "RemoveContainer" containerID="631bc24a6282ac61000fabbaa87b0e99fe9e10f69154cb35a2da53ee1d10c320"
Sep 15 21:54:11.204044 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:54:11.204027    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631bc24a6282ac61000fabbaa87b0e99fe9e10f69154cb35a2da53ee1d10c320\": container with ID starting with 631bc24a6282ac61000fabbaa87b0e99fe9e10f69154cb35a2da53ee1d10c320 not found: ID does not exist" containerID="631bc24a6282ac61000fabbaa87b0e99fe9e10f69154cb35a2da53ee1d10c320"
Sep 15 21:54:11.204108 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:11.204049    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631bc24a6282ac61000fabbaa87b0e99fe9e10f69154cb35a2da53ee1d10c320"} err="failed to get container status \"631bc24a6282ac61000fabbaa87b0e99fe9e10f69154cb35a2da53ee1d10c320\": rpc error: code = NotFound desc = could not find container \"631bc24a6282ac61000fabbaa87b0e99fe9e10f69154cb35a2da53ee1d10c320\": container with ID starting with 631bc24a6282ac61000fabbaa87b0e99fe9e10f69154cb35a2da53ee1d10c320 not found: ID does not exist"
Sep 15 21:54:11.204386 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:11.204366    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-850f22-openshift-default-5656cdb768-z854q"]
Sep 15 21:54:12.748638 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:12.748591    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" path="/var/lib/kubelet/pods/f48b61d5-9f9c-4527-b08c-c3fd2523bc98/volumes"
Sep 15 21:54:16.189368 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.189325    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"]
Sep 15 21:54:16.189895 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.189600    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb" podUID="2d75e36f-b1a1-4131-a27d-60989af5d770" containerName="payload-processing" containerID="cri-o://c7a6aed7144c17a81c643a045ef11346ddc2cc09f936015d81bc0ccfd91f83dc" gracePeriod=30
Sep 15 21:54:16.196852 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.196816    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"]
Sep 15 21:54:16.197343 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.197308    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx" podUID="2f9abe4b-1c48-421c-aed8-623679c0f32f" containerName="payload-pre-processing" containerID="cri-o://d93363455a48f98900d2600afd05f1a57481756f43ca9cf42094db304fd25da5" gracePeriod=30
Sep 15 21:54:16.341374 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.341326    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2"]
Sep 15 21:54:16.346055 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.346014    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8c1d2a-bp8n2"]
Sep 15 21:54:16.495216 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.495183    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"
Sep 15 21:54:16.499120 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.499087    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"
Sep 15 21:54:16.607468 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.607424    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk9cz\" (UniqueName: \"kubernetes.io/projected/2d75e36f-b1a1-4131-a27d-60989af5d770-kube-api-access-vk9cz\") pod \"2d75e36f-b1a1-4131-a27d-60989af5d770\" (UID: \"2d75e36f-b1a1-4131-a27d-60989af5d770\") "
Sep 15 21:54:16.607665 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.607482    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f9abe4b-1c48-421c-aed8-623679c0f32f-plugins-config-volume\") pod \"2f9abe4b-1c48-421c-aed8-623679c0f32f\" (UID: \"2f9abe4b-1c48-421c-aed8-623679c0f32f\") "
Sep 15 21:54:16.607665 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.607536    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d75e36f-b1a1-4131-a27d-60989af5d770-plugins-config-volume\") pod \"2d75e36f-b1a1-4131-a27d-60989af5d770\" (UID: \"2d75e36f-b1a1-4131-a27d-60989af5d770\") "
Sep 15 21:54:16.607665 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.607597    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ltwn\" (UniqueName: \"kubernetes.io/projected/2f9abe4b-1c48-421c-aed8-623679c0f32f-kube-api-access-7ltwn\") pod \"2f9abe4b-1c48-421c-aed8-623679c0f32f\" (UID: \"2f9abe4b-1c48-421c-aed8-623679c0f32f\") "
Sep 15 21:54:16.607962 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.607934    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9abe4b-1c48-421c-aed8-623679c0f32f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f9abe4b-1c48-421c-aed8-623679c0f32f" (UID: "2f9abe4b-1c48-421c-aed8-623679c0f32f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:54:16.608028 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.608006    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f9abe4b-1c48-421c-aed8-623679c0f32f-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:16.608028 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.608011    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d75e36f-b1a1-4131-a27d-60989af5d770-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d75e36f-b1a1-4131-a27d-60989af5d770" (UID: "2d75e36f-b1a1-4131-a27d-60989af5d770"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:54:16.610017 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.609982    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9abe4b-1c48-421c-aed8-623679c0f32f-kube-api-access-7ltwn" (OuterVolumeSpecName: "kube-api-access-7ltwn") pod "2f9abe4b-1c48-421c-aed8-623679c0f32f" (UID: "2f9abe4b-1c48-421c-aed8-623679c0f32f"). InnerVolumeSpecName "kube-api-access-7ltwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:54:16.610139 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.610028    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d75e36f-b1a1-4131-a27d-60989af5d770-kube-api-access-vk9cz" (OuterVolumeSpecName: "kube-api-access-vk9cz") pod "2d75e36f-b1a1-4131-a27d-60989af5d770" (UID: "2d75e36f-b1a1-4131-a27d-60989af5d770"). InnerVolumeSpecName "kube-api-access-vk9cz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:54:16.708647 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.708601    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d75e36f-b1a1-4131-a27d-60989af5d770-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:16.708647 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.708635    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ltwn\" (UniqueName: \"kubernetes.io/projected/2f9abe4b-1c48-421c-aed8-623679c0f32f-kube-api-access-7ltwn\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:16.708647 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.708645    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk9cz\" (UniqueName: \"kubernetes.io/projected/2d75e36f-b1a1-4131-a27d-60989af5d770-kube-api-access-vk9cz\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:16.748142 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:16.748052    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26ed83c0-588b-4f54-9dc4-027f8639621f" path="/var/lib/kubelet/pods/26ed83c0-588b-4f54-9dc4-027f8639621f/volumes"
Sep 15 21:54:17.222879 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.222844    2575 generic.go:356] "Generic (PLEG): container finished" podID="2d75e36f-b1a1-4131-a27d-60989af5d770" containerID="c7a6aed7144c17a81c643a045ef11346ddc2cc09f936015d81bc0ccfd91f83dc" exitCode=0
Sep 15 21:54:17.223383 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.222894    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"
Sep 15 21:54:17.223383 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.222933    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb" event={"ID":"2d75e36f-b1a1-4131-a27d-60989af5d770","Type":"ContainerDied","Data":"c7a6aed7144c17a81c643a045ef11346ddc2cc09f936015d81bc0ccfd91f83dc"}
Sep 15 21:54:17.223383 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.222985    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb" event={"ID":"2d75e36f-b1a1-4131-a27d-60989af5d770","Type":"ContainerDied","Data":"2ce239c20f06ef803bde4d40f397eeabfafc11a67c2852fc0dafda0054862ceb"}
Sep 15 21:54:17.223383 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.223010    2575 scope.go:117] "RemoveContainer" containerID="c7a6aed7144c17a81c643a045ef11346ddc2cc09f936015d81bc0ccfd91f83dc"
Sep 15 21:54:17.224665 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.224642    2575 generic.go:356] "Generic (PLEG): container finished" podID="2f9abe4b-1c48-421c-aed8-623679c0f32f" containerID="d93363455a48f98900d2600afd05f1a57481756f43ca9cf42094db304fd25da5" exitCode=0
Sep 15 21:54:17.224741 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.224687    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx" event={"ID":"2f9abe4b-1c48-421c-aed8-623679c0f32f","Type":"ContainerDied","Data":"d93363455a48f98900d2600afd05f1a57481756f43ca9cf42094db304fd25da5"}
Sep 15 21:54:17.224741 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.224704    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"
Sep 15 21:54:17.224741 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.224731    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx" event={"ID":"2f9abe4b-1c48-421c-aed8-623679c0f32f","Type":"ContainerDied","Data":"b01fbca0926ca5ccb5d0c2fefb166822c578dd2d5fda8544f105278d894276ed"}
Sep 15 21:54:17.233574 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.233551    2575 scope.go:117] "RemoveContainer" containerID="c7a6aed7144c17a81c643a045ef11346ddc2cc09f936015d81bc0ccfd91f83dc"
Sep 15 21:54:17.233994 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:54:17.233971    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a6aed7144c17a81c643a045ef11346ddc2cc09f936015d81bc0ccfd91f83dc\": container with ID starting with c7a6aed7144c17a81c643a045ef11346ddc2cc09f936015d81bc0ccfd91f83dc not found: ID does not exist" containerID="c7a6aed7144c17a81c643a045ef11346ddc2cc09f936015d81bc0ccfd91f83dc"
Sep 15 21:54:17.234089 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.234001    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a6aed7144c17a81c643a045ef11346ddc2cc09f936015d81bc0ccfd91f83dc"} err="failed to get container status \"c7a6aed7144c17a81c643a045ef11346ddc2cc09f936015d81bc0ccfd91f83dc\": rpc error: code = NotFound desc = could not find container \"c7a6aed7144c17a81c643a045ef11346ddc2cc09f936015d81bc0ccfd91f83dc\": container with ID starting with c7a6aed7144c17a81c643a045ef11346ddc2cc09f936015d81bc0ccfd91f83dc not found: ID does not exist"
Sep 15 21:54:17.234089 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.234022    2575 scope.go:117] "RemoveContainer" containerID="d93363455a48f98900d2600afd05f1a57481756f43ca9cf42094db304fd25da5"
Sep 15 21:54:17.243902 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.243740    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"]
Sep 15 21:54:17.243968 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.243938    2575 scope.go:117] "RemoveContainer" containerID="d93363455a48f98900d2600afd05f1a57481756f43ca9cf42094db304fd25da5"
Sep 15 21:54:17.244290 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:54:17.244271    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d93363455a48f98900d2600afd05f1a57481756f43ca9cf42094db304fd25da5\": container with ID starting with d93363455a48f98900d2600afd05f1a57481756f43ca9cf42094db304fd25da5 not found: ID does not exist" containerID="d93363455a48f98900d2600afd05f1a57481756f43ca9cf42094db304fd25da5"
Sep 15 21:54:17.244377 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.244296    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d93363455a48f98900d2600afd05f1a57481756f43ca9cf42094db304fd25da5"} err="failed to get container status \"d93363455a48f98900d2600afd05f1a57481756f43ca9cf42094db304fd25da5\": rpc error: code = NotFound desc = could not find container \"d93363455a48f98900d2600afd05f1a57481756f43ca9cf42094db304fd25da5\": container with ID starting with d93363455a48f98900d2600afd05f1a57481756f43ca9cf42094db304fd25da5 not found: ID does not exist"
Sep 15 21:54:17.247528 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.247492    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8c1d2a-779f98dd6b-sncfx"]
Sep 15 21:54:17.257431 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.257393    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"]
Sep 15 21:54:17.260930 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:17.260896    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8c1d2a-856857697c-stjlb"]
Sep 15 21:54:18.748295 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:18.748253    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d75e36f-b1a1-4131-a27d-60989af5d770" path="/var/lib/kubelet/pods/2d75e36f-b1a1-4131-a27d-60989af5d770/volumes"
Sep 15 21:54:18.748704 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:18.748644    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f9abe4b-1c48-421c-aed8-623679c0f32f" path="/var/lib/kubelet/pods/2f9abe4b-1c48-421c-aed8-623679c0f32f/volumes"
Sep 15 21:54:27.641472 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.641372    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55"]
Sep 15 21:54:27.642058 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.642036    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" containerName="istio-proxy"
Sep 15 21:54:27.642137 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.642062    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" containerName="istio-proxy"
Sep 15 21:54:27.642137 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.642093    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d75e36f-b1a1-4131-a27d-60989af5d770" containerName="payload-processing"
Sep 15 21:54:27.642137 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.642102    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d75e36f-b1a1-4131-a27d-60989af5d770" containerName="payload-processing"
Sep 15 21:54:27.642137 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.642112    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26ed83c0-588b-4f54-9dc4-027f8639621f" containerName="revoke-keys"
Sep 15 21:54:27.642137 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.642121    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="26ed83c0-588b-4f54-9dc4-027f8639621f" containerName="revoke-keys"
Sep 15 21:54:27.642137 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.642138    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f9abe4b-1c48-421c-aed8-623679c0f32f" containerName="payload-pre-processing"
Sep 15 21:54:27.642432 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.642146    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f9abe4b-1c48-421c-aed8-623679c0f32f" containerName="payload-pre-processing"
Sep 15 21:54:27.642432 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.642258    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d75e36f-b1a1-4131-a27d-60989af5d770" containerName="payload-processing"
Sep 15 21:54:27.642432 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.642272    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" containerName="istio-proxy"
Sep 15 21:54:27.642432 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.642283    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f9abe4b-1c48-421c-aed8-623679c0f32f" containerName="payload-pre-processing"
Sep 15 21:54:27.642432 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.642295    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="26ed83c0-588b-4f54-9dc4-027f8639621f" containerName="revoke-keys"
Sep 15 21:54:27.647273 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.647244    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55"
Sep 15 21:54:27.649620 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.649595    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:54:27.649767 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.649595    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 21:54:27.654860 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.654826    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55"]
Sep 15 21:54:27.727540 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.727487    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c4f7dc-17ca-46e0-8154-39ee0d963ca5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55\" (UID: \"92c4f7dc-17ca-46e0-8154-39ee0d963ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55"
Sep 15 21:54:27.828960 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.828908    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c4f7dc-17ca-46e0-8154-39ee0d963ca5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55\" (UID: \"92c4f7dc-17ca-46e0-8154-39ee0d963ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55"
Sep 15 21:54:27.829674 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.829652    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c4f7dc-17ca-46e0-8154-39ee0d963ca5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55\" (UID: \"92c4f7dc-17ca-46e0-8154-39ee0d963ca5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55"
Sep 15 21:54:27.960244 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:27.960142    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55"
Sep 15 21:54:28.237918 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:28.237832    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55"]
Sep 15 21:54:28.241083 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:54:28.241043    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c4f7dc_17ca_46e0_8154_39ee0d963ca5.slice/crio-8f5b229be3912a5594c3fe1f5d6aca2f640513a2fe107de3df561ac24fc60ae3 WatchSource:0}: Error finding container 8f5b229be3912a5594c3fe1f5d6aca2f640513a2fe107de3df561ac24fc60ae3: Status 404 returned error can't find the container with id 8f5b229be3912a5594c3fe1f5d6aca2f640513a2fe107de3df561ac24fc60ae3
Sep 15 21:54:28.295837 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:28.295802    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55" event={"ID":"92c4f7dc-17ca-46e0-8154-39ee0d963ca5","Type":"ContainerStarted","Data":"8f5b229be3912a5594c3fe1f5d6aca2f640513a2fe107de3df561ac24fc60ae3"}
Sep 15 21:54:29.303215 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:29.303115    2575 generic.go:356] "Generic (PLEG): container finished" podID="92c4f7dc-17ca-46e0-8154-39ee0d963ca5" containerID="21be41339e605d1a89ad12f184383977cc201aadcc65cbf506a4f87461ec8e6c" exitCode=0
Sep 15 21:54:29.303582 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:29.303220    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55" event={"ID":"92c4f7dc-17ca-46e0-8154-39ee0d963ca5","Type":"ContainerDied","Data":"21be41339e605d1a89ad12f184383977cc201aadcc65cbf506a4f87461ec8e6c"}
Sep 15 21:54:30.449342 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:30.449312    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55"
Sep 15 21:54:30.553036 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:30.552997    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c4f7dc-17ca-46e0-8154-39ee0d963ca5-maas-api-service-ca\") pod \"92c4f7dc-17ca-46e0-8154-39ee0d963ca5\" (UID: \"92c4f7dc-17ca-46e0-8154-39ee0d963ca5\") "
Sep 15 21:54:30.553375 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:30.553351    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c4f7dc-17ca-46e0-8154-39ee0d963ca5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92c4f7dc-17ca-46e0-8154-39ee0d963ca5" (UID: "92c4f7dc-17ca-46e0-8154-39ee0d963ca5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:54:30.654001 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:30.653965    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c4f7dc-17ca-46e0-8154-39ee0d963ca5-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:31.315640 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:31.315608    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55"
Sep 15 21:54:31.315877 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:31.315653    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55" event={"ID":"92c4f7dc-17ca-46e0-8154-39ee0d963ca5","Type":"ContainerDied","Data":"8f5b229be3912a5594c3fe1f5d6aca2f640513a2fe107de3df561ac24fc60ae3"}
Sep 15 21:54:31.315877 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:31.315707    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f5b229be3912a5594c3fe1f5d6aca2f640513a2fe107de3df561ac24fc60ae3"
Sep 15 21:54:32.526285 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:32.526250    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"]
Sep 15 21:54:35.041488 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:35.041434    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:54:43.284913 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.284874    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"]
Sep 15 21:54:43.285414 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.285138    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p" podUID="c882ebae-b6ef-4b97-8162-0dc3e3ee03aa" containerName="payload-processing" containerID="cri-o://da6e02d70cb03f8b802143980e660cf5d62808cbcafac94546296f500db29cea" gracePeriod=30
Sep 15 21:54:43.294151 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.293344    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"]
Sep 15 21:54:43.294151 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.293624    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz" podUID="b7b0d4c2-d78a-4580-821d-19ed1a49f238" containerName="payload-pre-processing" containerID="cri-o://0619e1207ebef1195049ccadc768b4090114f6bda722c21c35c8031dc3aecfcd" gracePeriod=30
Sep 15 21:54:43.450179 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.450116    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55"]
Sep 15 21:54:43.456525 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.456495    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-360e43-fmr55"]
Sep 15 21:54:43.575315 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.575288    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"
Sep 15 21:54:43.578487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.578462    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"
Sep 15 21:54:43.594670 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.594633    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbsl5\" (UniqueName: \"kubernetes.io/projected/b7b0d4c2-d78a-4580-821d-19ed1a49f238-kube-api-access-wbsl5\") pod \"b7b0d4c2-d78a-4580-821d-19ed1a49f238\" (UID: \"b7b0d4c2-d78a-4580-821d-19ed1a49f238\") "
Sep 15 21:54:43.594951 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.594728    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gk67\" (UniqueName: \"kubernetes.io/projected/c882ebae-b6ef-4b97-8162-0dc3e3ee03aa-kube-api-access-5gk67\") pod \"c882ebae-b6ef-4b97-8162-0dc3e3ee03aa\" (UID: \"c882ebae-b6ef-4b97-8162-0dc3e3ee03aa\") "
Sep 15 21:54:43.594951 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.594871    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c882ebae-b6ef-4b97-8162-0dc3e3ee03aa-plugins-config-volume\") pod \"c882ebae-b6ef-4b97-8162-0dc3e3ee03aa\" (UID: \"c882ebae-b6ef-4b97-8162-0dc3e3ee03aa\") "
Sep 15 21:54:43.594951 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.594909    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7b0d4c2-d78a-4580-821d-19ed1a49f238-plugins-config-volume\") pod \"b7b0d4c2-d78a-4580-821d-19ed1a49f238\" (UID: \"b7b0d4c2-d78a-4580-821d-19ed1a49f238\") "
Sep 15 21:54:43.595405 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.595350    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c882ebae-b6ef-4b97-8162-0dc3e3ee03aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c882ebae-b6ef-4b97-8162-0dc3e3ee03aa" (UID: "c882ebae-b6ef-4b97-8162-0dc3e3ee03aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:54:43.595632 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.595608    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b0d4c2-d78a-4580-821d-19ed1a49f238-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7b0d4c2-d78a-4580-821d-19ed1a49f238" (UID: "b7b0d4c2-d78a-4580-821d-19ed1a49f238"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:54:43.597156 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.597131    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b0d4c2-d78a-4580-821d-19ed1a49f238-kube-api-access-wbsl5" (OuterVolumeSpecName: "kube-api-access-wbsl5") pod "b7b0d4c2-d78a-4580-821d-19ed1a49f238" (UID: "b7b0d4c2-d78a-4580-821d-19ed1a49f238"). InnerVolumeSpecName "kube-api-access-wbsl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:54:43.597487 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.597465    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c882ebae-b6ef-4b97-8162-0dc3e3ee03aa-kube-api-access-5gk67" (OuterVolumeSpecName: "kube-api-access-5gk67") pod "c882ebae-b6ef-4b97-8162-0dc3e3ee03aa" (UID: "c882ebae-b6ef-4b97-8162-0dc3e3ee03aa"). InnerVolumeSpecName "kube-api-access-5gk67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:54:43.696923 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.696870    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c882ebae-b6ef-4b97-8162-0dc3e3ee03aa-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:43.696923 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.696913    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7b0d4c2-d78a-4580-821d-19ed1a49f238-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:43.696923 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.696927    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbsl5\" (UniqueName: \"kubernetes.io/projected/b7b0d4c2-d78a-4580-821d-19ed1a49f238-kube-api-access-wbsl5\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:43.697186 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:43.696938    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gk67\" (UniqueName: \"kubernetes.io/projected/c882ebae-b6ef-4b97-8162-0dc3e3ee03aa-kube-api-access-5gk67\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:44.388410 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.388365    2575 generic.go:356] "Generic (PLEG): container finished" podID="c882ebae-b6ef-4b97-8162-0dc3e3ee03aa" containerID="da6e02d70cb03f8b802143980e660cf5d62808cbcafac94546296f500db29cea" exitCode=0
Sep 15 21:54:44.389001 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.388417    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"
Sep 15 21:54:44.389001 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.388452    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p" event={"ID":"c882ebae-b6ef-4b97-8162-0dc3e3ee03aa","Type":"ContainerDied","Data":"da6e02d70cb03f8b802143980e660cf5d62808cbcafac94546296f500db29cea"}
Sep 15 21:54:44.389001 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.388515    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p" event={"ID":"c882ebae-b6ef-4b97-8162-0dc3e3ee03aa","Type":"ContainerDied","Data":"170069c64ca790aec2aa503ec248c791b648ebf9fe6090451223dacbadd5d0c7"}
Sep 15 21:54:44.389001 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.388547    2575 scope.go:117] "RemoveContainer" containerID="da6e02d70cb03f8b802143980e660cf5d62808cbcafac94546296f500db29cea"
Sep 15 21:54:44.390019 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.389992    2575 generic.go:356] "Generic (PLEG): container finished" podID="b7b0d4c2-d78a-4580-821d-19ed1a49f238" containerID="0619e1207ebef1195049ccadc768b4090114f6bda722c21c35c8031dc3aecfcd" exitCode=0
Sep 15 21:54:44.390089 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.390045    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"
Sep 15 21:54:44.390138 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.390114    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz" event={"ID":"b7b0d4c2-d78a-4580-821d-19ed1a49f238","Type":"ContainerDied","Data":"0619e1207ebef1195049ccadc768b4090114f6bda722c21c35c8031dc3aecfcd"}
Sep 15 21:54:44.390178 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.390167    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz" event={"ID":"b7b0d4c2-d78a-4580-821d-19ed1a49f238","Type":"ContainerDied","Data":"a8d2fd8d1d91a6544f98248e271d1a085af0a61b7eb5713169592b21ac759ef3"}
Sep 15 21:54:44.400577 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.400457    2575 scope.go:117] "RemoveContainer" containerID="da6e02d70cb03f8b802143980e660cf5d62808cbcafac94546296f500db29cea"
Sep 15 21:54:44.401008 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:54:44.400984    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da6e02d70cb03f8b802143980e660cf5d62808cbcafac94546296f500db29cea\": container with ID starting with da6e02d70cb03f8b802143980e660cf5d62808cbcafac94546296f500db29cea not found: ID does not exist" containerID="da6e02d70cb03f8b802143980e660cf5d62808cbcafac94546296f500db29cea"
Sep 15 21:54:44.401089 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.401017    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da6e02d70cb03f8b802143980e660cf5d62808cbcafac94546296f500db29cea"} err="failed to get container status \"da6e02d70cb03f8b802143980e660cf5d62808cbcafac94546296f500db29cea\": rpc error: code = NotFound desc = could not find container \"da6e02d70cb03f8b802143980e660cf5d62808cbcafac94546296f500db29cea\": container with ID starting with da6e02d70cb03f8b802143980e660cf5d62808cbcafac94546296f500db29cea not found: ID does not exist"
Sep 15 21:54:44.401089 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.401036    2575 scope.go:117] "RemoveContainer" containerID="0619e1207ebef1195049ccadc768b4090114f6bda722c21c35c8031dc3aecfcd"
Sep 15 21:54:44.410367 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.410345    2575 scope.go:117] "RemoveContainer" containerID="0619e1207ebef1195049ccadc768b4090114f6bda722c21c35c8031dc3aecfcd"
Sep 15 21:54:44.410721 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:54:44.410699    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0619e1207ebef1195049ccadc768b4090114f6bda722c21c35c8031dc3aecfcd\": container with ID starting with 0619e1207ebef1195049ccadc768b4090114f6bda722c21c35c8031dc3aecfcd not found: ID does not exist" containerID="0619e1207ebef1195049ccadc768b4090114f6bda722c21c35c8031dc3aecfcd"
Sep 15 21:54:44.410809 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.410734    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0619e1207ebef1195049ccadc768b4090114f6bda722c21c35c8031dc3aecfcd"} err="failed to get container status \"0619e1207ebef1195049ccadc768b4090114f6bda722c21c35c8031dc3aecfcd\": rpc error: code = NotFound desc = could not find container \"0619e1207ebef1195049ccadc768b4090114f6bda722c21c35c8031dc3aecfcd\": container with ID starting with 0619e1207ebef1195049ccadc768b4090114f6bda722c21c35c8031dc3aecfcd not found: ID does not exist"
Sep 15 21:54:44.417536 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.417506    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"]
Sep 15 21:54:44.419637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.419611    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-360e43-56bdfb8d6b-tnxcz"]
Sep 15 21:54:44.429489 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.429453    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"]
Sep 15 21:54:44.433932 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.433891    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-360e43-785576c97c-jv62p"]
Sep 15 21:54:44.748177 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.748091    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92c4f7dc-17ca-46e0-8154-39ee0d963ca5" path="/var/lib/kubelet/pods/92c4f7dc-17ca-46e0-8154-39ee0d963ca5/volumes"
Sep 15 21:54:44.748520 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.748505    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7b0d4c2-d78a-4580-821d-19ed1a49f238" path="/var/lib/kubelet/pods/b7b0d4c2-d78a-4580-821d-19ed1a49f238/volumes"
Sep 15 21:54:44.748849 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:44.748836    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c882ebae-b6ef-4b97-8162-0dc3e3ee03aa" path="/var/lib/kubelet/pods/c882ebae-b6ef-4b97-8162-0dc3e3ee03aa/volumes"
Sep 15 21:54:47.547808 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.547747    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4" podUID="31162923-010d-46fc-8522-f51eb79ab9f4" containerName="main" containerID="cri-o://bedeb4e4b664a4cabc984460c7131d8248bc64937b9fe28b3ef4b54973ecbcf1" gracePeriod=45
Sep 15 21:54:47.812187 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.812109    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:54:47.838430 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.838392    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-home\") pod \"31162923-010d-46fc-8522-f51eb79ab9f4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") "
Sep 15 21:54:47.838606 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.838527    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-htjbf\" (UniqueName: \"kubernetes.io/projected/31162923-010d-46fc-8522-f51eb79ab9f4-kube-api-access-htjbf\") pod \"31162923-010d-46fc-8522-f51eb79ab9f4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") "
Sep 15 21:54:47.838606 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.838564    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-dshm\") pod \"31162923-010d-46fc-8522-f51eb79ab9f4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") "
Sep 15 21:54:47.838606 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.838604    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-model-cache\") pod \"31162923-010d-46fc-8522-f51eb79ab9f4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") "
Sep 15 21:54:47.838813 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.838677    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31162923-010d-46fc-8522-f51eb79ab9f4-tls-certs\") pod \"31162923-010d-46fc-8522-f51eb79ab9f4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") "
Sep 15 21:54:47.838813 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.838705    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-tmp-dir\") pod \"31162923-010d-46fc-8522-f51eb79ab9f4\" (UID: \"31162923-010d-46fc-8522-f51eb79ab9f4\") "
Sep 15 21:54:47.838813 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.838703    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-home" (OuterVolumeSpecName: "home") pod "31162923-010d-46fc-8522-f51eb79ab9f4" (UID: "31162923-010d-46fc-8522-f51eb79ab9f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:54:47.839030 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.838995    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "31162923-010d-46fc-8522-f51eb79ab9f4" (UID: "31162923-010d-46fc-8522-f51eb79ab9f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:54:47.839188 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.839092    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:47.839262 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.839181    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31162923-010d-46fc-8522-f51eb79ab9f4" (UID: "31162923-010d-46fc-8522-f51eb79ab9f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:54:47.841090 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.841068    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31162923-010d-46fc-8522-f51eb79ab9f4-kube-api-access-htjbf" (OuterVolumeSpecName: "kube-api-access-htjbf") pod "31162923-010d-46fc-8522-f51eb79ab9f4" (UID: "31162923-010d-46fc-8522-f51eb79ab9f4"). InnerVolumeSpecName "kube-api-access-htjbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:54:47.841429 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.841410    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-dshm" (OuterVolumeSpecName: "dshm") pod "31162923-010d-46fc-8522-f51eb79ab9f4" (UID: "31162923-010d-46fc-8522-f51eb79ab9f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:54:47.841519 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.841489    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31162923-010d-46fc-8522-f51eb79ab9f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31162923-010d-46fc-8522-f51eb79ab9f4" (UID: "31162923-010d-46fc-8522-f51eb79ab9f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:54:47.940310 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.940273    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31162923-010d-46fc-8522-f51eb79ab9f4-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:47.940310 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.940304    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:47.940310 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.940313    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htjbf\" (UniqueName: \"kubernetes.io/projected/31162923-010d-46fc-8522-f51eb79ab9f4-kube-api-access-htjbf\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:47.940627 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.940324    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:47.940627 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:47.940333    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31162923-010d-46fc-8522-f51eb79ab9f4-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:54:48.418113 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:48.418065    2575 generic.go:356] "Generic (PLEG): container finished" podID="31162923-010d-46fc-8522-f51eb79ab9f4" containerID="bedeb4e4b664a4cabc984460c7131d8248bc64937b9fe28b3ef4b54973ecbcf1" exitCode=0
Sep 15 21:54:48.418113 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:48.418105    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"
Sep 15 21:54:48.418379 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:48.418126    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4" event={"ID":"31162923-010d-46fc-8522-f51eb79ab9f4","Type":"ContainerDied","Data":"bedeb4e4b664a4cabc984460c7131d8248bc64937b9fe28b3ef4b54973ecbcf1"}
Sep 15 21:54:48.418379 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:48.418173    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4" event={"ID":"31162923-010d-46fc-8522-f51eb79ab9f4","Type":"ContainerDied","Data":"1fdcee0232ba402acdd000d12a4e1939a8ecee97e992552e77ab7c449bf3c60f"}
Sep 15 21:54:48.418379 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:48.418196    2575 scope.go:117] "RemoveContainer" containerID="bedeb4e4b664a4cabc984460c7131d8248bc64937b9fe28b3ef4b54973ecbcf1"
Sep 15 21:54:48.431526 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:48.431496    2575 scope.go:117] "RemoveContainer" containerID="bedeb4e4b664a4cabc984460c7131d8248bc64937b9fe28b3ef4b54973ecbcf1"
Sep 15 21:54:48.431982 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:54:48.431954    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bedeb4e4b664a4cabc984460c7131d8248bc64937b9fe28b3ef4b54973ecbcf1\": container with ID starting with bedeb4e4b664a4cabc984460c7131d8248bc64937b9fe28b3ef4b54973ecbcf1 not found: ID does not exist" containerID="bedeb4e4b664a4cabc984460c7131d8248bc64937b9fe28b3ef4b54973ecbcf1"
Sep 15 21:54:48.432052 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:48.431997    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bedeb4e4b664a4cabc984460c7131d8248bc64937b9fe28b3ef4b54973ecbcf1"} err="failed to get container status \"bedeb4e4b664a4cabc984460c7131d8248bc64937b9fe28b3ef4b54973ecbcf1\": rpc error: code = NotFound desc = could not find container \"bedeb4e4b664a4cabc984460c7131d8248bc64937b9fe28b3ef4b54973ecbcf1\": container with ID starting with bedeb4e4b664a4cabc984460c7131d8248bc64937b9fe28b3ef4b54973ecbcf1 not found: ID does not exist"
Sep 15 21:54:48.444384 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:48.444343    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"]
Sep 15 21:54:48.449974 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:48.449944    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-360e43/embedding-w0-360e43-kserve-75fbbd46f7-jzbn4"]
Sep 15 21:54:48.748283 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:54:48.748190    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31162923-010d-46fc-8522-f51eb79ab9f4" path="/var/lib/kubelet/pods/31162923-010d-46fc-8522-f51eb79ab9f4/volumes"
Sep 15 21:55:05.835303 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:05.835231    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:55:28.043519 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:28.043484    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:55:41.422100 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.422061    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7728-d18843"]
Sep 15 21:55:41.422495 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.422467    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7b0d4c2-d78a-4580-821d-19ed1a49f238" containerName="payload-pre-processing"
Sep 15 21:55:41.422495 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.422479    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7b0d4c2-d78a-4580-821d-19ed1a49f238" containerName="payload-pre-processing"
Sep 15 21:55:41.422495 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.422489    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" containerName="istio-proxy"
Sep 15 21:55:41.422495 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.422494    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" containerName="istio-proxy"
Sep 15 21:55:41.422637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.422508    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92c4f7dc-17ca-46e0-8154-39ee0d963ca5" containerName="revoke-keys"
Sep 15 21:55:41.422637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.422514    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="92c4f7dc-17ca-46e0-8154-39ee0d963ca5" containerName="revoke-keys"
Sep 15 21:55:41.422637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.422524    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c882ebae-b6ef-4b97-8162-0dc3e3ee03aa" containerName="payload-processing"
Sep 15 21:55:41.422637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.422530    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c882ebae-b6ef-4b97-8162-0dc3e3ee03aa" containerName="payload-processing"
Sep 15 21:55:41.422637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.422538    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31162923-010d-46fc-8522-f51eb79ab9f4" containerName="main"
Sep 15 21:55:41.422637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.422544    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="31162923-010d-46fc-8522-f51eb79ab9f4" containerName="main"
Sep 15 21:55:41.422637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.422616    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7b0d4c2-d78a-4580-821d-19ed1a49f238" containerName="payload-pre-processing"
Sep 15 21:55:41.422637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.422625    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="31162923-010d-46fc-8522-f51eb79ab9f4" containerName="main"
Sep 15 21:55:41.422637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.422632    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f48b61d5-9f9c-4527-b08c-c3fd2523bc98" containerName="istio-proxy"
Sep 15 21:55:41.422637 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.422639    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="92c4f7dc-17ca-46e0-8154-39ee0d963ca5" containerName="revoke-keys"
Sep 15 21:55:41.422979 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.422647    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c882ebae-b6ef-4b97-8162-0dc3e3ee03aa" containerName="payload-processing"
Sep 15 21:55:41.425833 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.425815    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7728-d18843"
Sep 15 21:55:41.428253 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.428224    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2lhf2\""
Sep 15 21:55:41.431247 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.431218    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7728-d18843"]
Sep 15 21:55:41.589260 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.589219    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2zbr\" (UniqueName: \"kubernetes.io/projected/bda9a9c9-82d8-48dc-8ee7-363b11775cd5-kube-api-access-p2zbr\") pod \"test-curl-7728-d18843\" (UID: \"bda9a9c9-82d8-48dc-8ee7-363b11775cd5\") " pod="opendatahub/test-curl-7728-d18843"
Sep 15 21:55:41.690423 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.690330    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2zbr\" (UniqueName: \"kubernetes.io/projected/bda9a9c9-82d8-48dc-8ee7-363b11775cd5-kube-api-access-p2zbr\") pod \"test-curl-7728-d18843\" (UID: \"bda9a9c9-82d8-48dc-8ee7-363b11775cd5\") " pod="opendatahub/test-curl-7728-d18843"
Sep 15 21:55:41.702323 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.702286    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2zbr\" (UniqueName: \"kubernetes.io/projected/bda9a9c9-82d8-48dc-8ee7-363b11775cd5-kube-api-access-p2zbr\") pod \"test-curl-7728-d18843\" (UID: \"bda9a9c9-82d8-48dc-8ee7-363b11775cd5\") " pod="opendatahub/test-curl-7728-d18843"
Sep 15 21:55:41.736603 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:41.736559    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7728-d18843"
Sep 15 21:55:42.007975 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:42.007925    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7728-d18843"]
Sep 15 21:55:42.010404 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:55:42.010372    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbda9a9c9_82d8_48dc_8ee7_363b11775cd5.slice/crio-0f6d57eec3685706cdb3e01483f614ec0c14a469c8f6ab2f27bf08e884ec6233 WatchSource:0}: Error finding container 0f6d57eec3685706cdb3e01483f614ec0c14a469c8f6ab2f27bf08e884ec6233: Status 404 returned error can't find the container with id 0f6d57eec3685706cdb3e01483f614ec0c14a469c8f6ab2f27bf08e884ec6233
Sep 15 21:55:42.726127 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:42.726093    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7728-d18843" event={"ID":"bda9a9c9-82d8-48dc-8ee7-363b11775cd5","Type":"ContainerStarted","Data":"87c07e40774a82d6b74dc5b5081498f22184639fbe7d1217079d52bb9d029a40"}
Sep 15 21:55:42.726549 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:42.726150    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7728-d18843" event={"ID":"bda9a9c9-82d8-48dc-8ee7-363b11775cd5","Type":"ContainerStarted","Data":"0f6d57eec3685706cdb3e01483f614ec0c14a469c8f6ab2f27bf08e884ec6233"}
Sep 15 21:55:42.741582 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:42.741521    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7728-d18843" podStartSLOduration=1.741504924 podStartE2EDuration="1.741504924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:55:42.740619593 +0000 UTC m=+1780.552628046" watchObservedRunningTime="2026-09-15 21:55:42.741504924 +0000 UTC m=+1780.553513368"
Sep 15 21:55:43.359948 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:43.359911    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7728-d18843"]
Sep 15 21:55:43.364228 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:43.364200    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7728-d18843"]
Sep 15 21:55:43.731494 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:55:43.731454    2575 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7728-d18843" container="test-curl-7728-d18843"
Sep 15 21:55:44.736537 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:44.736476    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7728-d18843" podUID="bda9a9c9-82d8-48dc-8ee7-363b11775cd5" containerName="test-curl-7728-d18843" containerID="cri-o://87c07e40774a82d6b74dc5b5081498f22184639fbe7d1217079d52bb9d029a40" gracePeriod=2
Sep 15 21:55:44.738958 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:55:44.738926    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7728-d18843\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="bda9a9c9-82d8-48dc-8ee7-363b11775cd5" pod="opendatahub/test-curl-7728-d18843"
Sep 15 21:55:44.948354 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:44.948318    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn"]
Sep 15 21:55:44.948798 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:44.948783    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda9a9c9-82d8-48dc-8ee7-363b11775cd5" containerName="test-curl-7728-d18843"
Sep 15 21:55:44.948853 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:44.948800    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda9a9c9-82d8-48dc-8ee7-363b11775cd5" containerName="test-curl-7728-d18843"
Sep 15 21:55:44.948904 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:44.948894    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bda9a9c9-82d8-48dc-8ee7-363b11775cd5" containerName="test-curl-7728-d18843"
Sep 15 21:55:44.951893 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:44.951874    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn"
Sep 15 21:55:44.954085 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:44.954066    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 21:55:44.954203 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:44.954064    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 21:55:44.958240 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:44.958204    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn"]
Sep 15 21:55:45.129837 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:45.129748    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d53b1de-13a4-4639-8cbc-e8f2d76a27e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn\" (UID: \"3d53b1de-13a4-4639-8cbc-e8f2d76a27e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn"
Sep 15 21:55:45.231370 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:45.231323    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d53b1de-13a4-4639-8cbc-e8f2d76a27e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn\" (UID: \"3d53b1de-13a4-4639-8cbc-e8f2d76a27e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn"
Sep 15 21:55:45.232054 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:45.232036    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d53b1de-13a4-4639-8cbc-e8f2d76a27e9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn\" (UID: \"3d53b1de-13a4-4639-8cbc-e8f2d76a27e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn"
Sep 15 21:55:45.264630 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:45.264585    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn"
Sep 15 21:55:45.537502 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:45.537476    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn"]
Sep 15 21:55:45.539840 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:55:45.539805    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d53b1de_13a4_4639_8cbc_e8f2d76a27e9.slice/crio-5f7e6dfa0f7c1cf83bc3be5d5c7651942df3e72895e7b7b382bfa0494dcc300a WatchSource:0}: Error finding container 5f7e6dfa0f7c1cf83bc3be5d5c7651942df3e72895e7b7b382bfa0494dcc300a: Status 404 returned error can't find the container with id 5f7e6dfa0f7c1cf83bc3be5d5c7651942df3e72895e7b7b382bfa0494dcc300a
Sep 15 21:55:45.743489 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:45.743454    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn" event={"ID":"3d53b1de-13a4-4639-8cbc-e8f2d76a27e9","Type":"ContainerStarted","Data":"5f7e6dfa0f7c1cf83bc3be5d5c7651942df3e72895e7b7b382bfa0494dcc300a"}
Sep 15 21:55:46.117893 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:46.117859    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"]
Sep 15 21:55:46.754129 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:46.754100    2575 generic.go:356] "Generic (PLEG): container finished" podID="3d53b1de-13a4-4639-8cbc-e8f2d76a27e9" containerID="4b8720e01c42e3b18acbed79e68df7967ac31c0cd243f7e19b04301b5d82ff1f" exitCode=0
Sep 15 21:55:46.754564 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:46.754181    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn" event={"ID":"3d53b1de-13a4-4639-8cbc-e8f2d76a27e9","Type":"ContainerDied","Data":"4b8720e01c42e3b18acbed79e68df7967ac31c0cd243f7e19b04301b5d82ff1f"}
Sep 15 21:55:46.914419 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:46.914393    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7728-d18843"
Sep 15 21:55:46.950467 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:46.950434    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2zbr\" (UniqueName: \"kubernetes.io/projected/bda9a9c9-82d8-48dc-8ee7-363b11775cd5-kube-api-access-p2zbr\") pod \"bda9a9c9-82d8-48dc-8ee7-363b11775cd5\" (UID: \"bda9a9c9-82d8-48dc-8ee7-363b11775cd5\") "
Sep 15 21:55:46.952707 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:46.952677    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda9a9c9-82d8-48dc-8ee7-363b11775cd5-kube-api-access-p2zbr" (OuterVolumeSpecName: "kube-api-access-p2zbr") pod "bda9a9c9-82d8-48dc-8ee7-363b11775cd5" (UID: "bda9a9c9-82d8-48dc-8ee7-363b11775cd5"). InnerVolumeSpecName "kube-api-access-p2zbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:55:47.051974 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:47.051879    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2zbr\" (UniqueName: \"kubernetes.io/projected/bda9a9c9-82d8-48dc-8ee7-363b11775cd5-kube-api-access-p2zbr\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:55:47.762005 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:47.761966    2575 generic.go:356] "Generic (PLEG): container finished" podID="bda9a9c9-82d8-48dc-8ee7-363b11775cd5" containerID="87c07e40774a82d6b74dc5b5081498f22184639fbe7d1217079d52bb9d029a40" exitCode=137
Sep 15 21:55:47.762428 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:47.762031    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7728-d18843"
Sep 15 21:55:47.762650 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:47.762066    2575 scope.go:117] "RemoveContainer" containerID="87c07e40774a82d6b74dc5b5081498f22184639fbe7d1217079d52bb9d029a40"
Sep 15 21:55:47.772968 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:55:47.772931    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7728-d18843\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="bda9a9c9-82d8-48dc-8ee7-363b11775cd5" pod="opendatahub/test-curl-7728-d18843"
Sep 15 21:55:47.773300 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:47.773285    2575 scope.go:117] "RemoveContainer" containerID="87c07e40774a82d6b74dc5b5081498f22184639fbe7d1217079d52bb9d029a40"
Sep 15 21:55:47.773637 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:55:47.773615    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c07e40774a82d6b74dc5b5081498f22184639fbe7d1217079d52bb9d029a40\": container with ID starting with 87c07e40774a82d6b74dc5b5081498f22184639fbe7d1217079d52bb9d029a40 not found: ID does not exist" containerID="87c07e40774a82d6b74dc5b5081498f22184639fbe7d1217079d52bb9d029a40"
Sep 15 21:55:47.773710 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:47.773660    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c07e40774a82d6b74dc5b5081498f22184639fbe7d1217079d52bb9d029a40"} err="failed to get container status \"87c07e40774a82d6b74dc5b5081498f22184639fbe7d1217079d52bb9d029a40\": rpc error: code = NotFound desc = could not find container \"87c07e40774a82d6b74dc5b5081498f22184639fbe7d1217079d52bb9d029a40\": container with ID starting with 87c07e40774a82d6b74dc5b5081498f22184639fbe7d1217079d52bb9d029a40 not found: ID does not exist"
Sep 15 21:55:47.901446 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:47.901414    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn"
Sep 15 21:55:47.917799 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:55:47.917731    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7728-d18843\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="bda9a9c9-82d8-48dc-8ee7-363b11775cd5" pod="opendatahub/test-curl-7728-d18843"
Sep 15 21:55:47.959389 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:47.959356    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d53b1de-13a4-4639-8cbc-e8f2d76a27e9-maas-api-service-ca\") pod \"3d53b1de-13a4-4639-8cbc-e8f2d76a27e9\" (UID: \"3d53b1de-13a4-4639-8cbc-e8f2d76a27e9\") "
Sep 15 21:55:47.959807 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:47.959773    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d53b1de-13a4-4639-8cbc-e8f2d76a27e9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d53b1de-13a4-4639-8cbc-e8f2d76a27e9" (UID: "3d53b1de-13a4-4639-8cbc-e8f2d76a27e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:55:48.061079 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:48.060990    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d53b1de-13a4-4639-8cbc-e8f2d76a27e9-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:55:48.747534 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:48.747498    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bda9a9c9-82d8-48dc-8ee7-363b11775cd5" path="/var/lib/kubelet/pods/bda9a9c9-82d8-48dc-8ee7-363b11775cd5/volumes"
Sep 15 21:55:48.769829 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:48.769800    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn"
Sep 15 21:55:48.769829 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:48.769818    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn" event={"ID":"3d53b1de-13a4-4639-8cbc-e8f2d76a27e9","Type":"ContainerDied","Data":"5f7e6dfa0f7c1cf83bc3be5d5c7651942df3e72895e7b7b382bfa0494dcc300a"}
Sep 15 21:55:48.770339 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:48.769874    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f7e6dfa0f7c1cf83bc3be5d5c7651942df3e72895e7b7b382bfa0494dcc300a"
Sep 15 21:55:49.156593 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:49.156560    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:55:49.819198 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:49.819158    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"]
Sep 15 21:55:49.826334 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:49.826301    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"]
Sep 15 21:55:49.847827 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:55:49.847789    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"]
Sep 15 21:56:01.138423 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.138314    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2" podUID="def6dd16-fb1a-4d0f-9881-51d147f35833" containerName="main" containerID="cri-o://001e34ed5b24284dc198b821e44a9a28e0ab199fb7edf8cf5cd42e2b59ed525b" gracePeriod=45
Sep 15 21:56:01.232412 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.232379    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn"]
Sep 15 21:56:01.236603 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.236577    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-56e305-kj5bn"]
Sep 15 21:56:01.390734 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.390666    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:56:01.496774 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.496732    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-tmp-dir\") pod \"def6dd16-fb1a-4d0f-9881-51d147f35833\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") "
Sep 15 21:56:01.496963 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.496849    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-dshm\") pod \"def6dd16-fb1a-4d0f-9881-51d147f35833\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") "
Sep 15 21:56:01.496963 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.496881    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-model-cache\") pod \"def6dd16-fb1a-4d0f-9881-51d147f35833\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") "
Sep 15 21:56:01.496963 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.496921    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def6dd16-fb1a-4d0f-9881-51d147f35833-tls-certs\") pod \"def6dd16-fb1a-4d0f-9881-51d147f35833\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") "
Sep 15 21:56:01.496963 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.496945    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-home\") pod \"def6dd16-fb1a-4d0f-9881-51d147f35833\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") "
Sep 15 21:56:01.497133 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.496974    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckq6q\" (UniqueName: \"kubernetes.io/projected/def6dd16-fb1a-4d0f-9881-51d147f35833-kube-api-access-ckq6q\") pod \"def6dd16-fb1a-4d0f-9881-51d147f35833\" (UID: \"def6dd16-fb1a-4d0f-9881-51d147f35833\") "
Sep 15 21:56:01.497133 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.497070    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "def6dd16-fb1a-4d0f-9881-51d147f35833" (UID: "def6dd16-fb1a-4d0f-9881-51d147f35833"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:56:01.497210 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.497129    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-model-cache" (OuterVolumeSpecName: "model-cache") pod "def6dd16-fb1a-4d0f-9881-51d147f35833" (UID: "def6dd16-fb1a-4d0f-9881-51d147f35833"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:56:01.497262 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.497234    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-home" (OuterVolumeSpecName: "home") pod "def6dd16-fb1a-4d0f-9881-51d147f35833" (UID: "def6dd16-fb1a-4d0f-9881-51d147f35833"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:56:01.497360 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.497339    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:01.497360 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.497361    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:01.497514 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.497388    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:01.499325 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.499298    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-dshm" (OuterVolumeSpecName: "dshm") pod "def6dd16-fb1a-4d0f-9881-51d147f35833" (UID: "def6dd16-fb1a-4d0f-9881-51d147f35833"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:56:01.499425 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.499331    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def6dd16-fb1a-4d0f-9881-51d147f35833-kube-api-access-ckq6q" (OuterVolumeSpecName: "kube-api-access-ckq6q") pod "def6dd16-fb1a-4d0f-9881-51d147f35833" (UID: "def6dd16-fb1a-4d0f-9881-51d147f35833"). InnerVolumeSpecName "kube-api-access-ckq6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:56:01.499425 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.499350    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/def6dd16-fb1a-4d0f-9881-51d147f35833-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "def6dd16-fb1a-4d0f-9881-51d147f35833" (UID: "def6dd16-fb1a-4d0f-9881-51d147f35833"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:56:01.598745 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.598706    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/def6dd16-fb1a-4d0f-9881-51d147f35833-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:01.598745 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.598735    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckq6q\" (UniqueName: \"kubernetes.io/projected/def6dd16-fb1a-4d0f-9881-51d147f35833-kube-api-access-ckq6q\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:01.598745 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.598745    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/def6dd16-fb1a-4d0f-9881-51d147f35833-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:01.845642 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.845613    2575 generic.go:356] "Generic (PLEG): container finished" podID="def6dd16-fb1a-4d0f-9881-51d147f35833" containerID="001e34ed5b24284dc198b821e44a9a28e0ab199fb7edf8cf5cd42e2b59ed525b" exitCode=0
Sep 15 21:56:01.845841 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.845660    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"
Sep 15 21:56:01.845841 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.845702    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2" event={"ID":"def6dd16-fb1a-4d0f-9881-51d147f35833","Type":"ContainerDied","Data":"001e34ed5b24284dc198b821e44a9a28e0ab199fb7edf8cf5cd42e2b59ed525b"}
Sep 15 21:56:01.845841 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.845746    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2" event={"ID":"def6dd16-fb1a-4d0f-9881-51d147f35833","Type":"ContainerDied","Data":"aa901ec2cd443a3f523ff6cfcaea5b08f3c8128f47f525896e8fb805cb497491"}
Sep 15 21:56:01.845841 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.845780    2575 scope.go:117] "RemoveContainer" containerID="001e34ed5b24284dc198b821e44a9a28e0ab199fb7edf8cf5cd42e2b59ed525b"
Sep 15 21:56:01.855372 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.855353    2575 scope.go:117] "RemoveContainer" containerID="001e34ed5b24284dc198b821e44a9a28e0ab199fb7edf8cf5cd42e2b59ed525b"
Sep 15 21:56:01.855669 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:56:01.855650    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"001e34ed5b24284dc198b821e44a9a28e0ab199fb7edf8cf5cd42e2b59ed525b\": container with ID starting with 001e34ed5b24284dc198b821e44a9a28e0ab199fb7edf8cf5cd42e2b59ed525b not found: ID does not exist" containerID="001e34ed5b24284dc198b821e44a9a28e0ab199fb7edf8cf5cd42e2b59ed525b"
Sep 15 21:56:01.855723 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.855678    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"001e34ed5b24284dc198b821e44a9a28e0ab199fb7edf8cf5cd42e2b59ed525b"} err="failed to get container status \"001e34ed5b24284dc198b821e44a9a28e0ab199fb7edf8cf5cd42e2b59ed525b\": rpc error: code = NotFound desc = could not find container \"001e34ed5b24284dc198b821e44a9a28e0ab199fb7edf8cf5cd42e2b59ed525b\": container with ID starting with 001e34ed5b24284dc198b821e44a9a28e0ab199fb7edf8cf5cd42e2b59ed525b not found: ID does not exist"
Sep 15 21:56:01.881463 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.881416    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"]
Sep 15 21:56:01.897965 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:01.897925    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/auth-test-model-e7a648-kserve-d7ffd659-6t4v2"]
Sep 15 21:56:02.750170 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:02.750133    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d53b1de-13a4-4639-8cbc-e8f2d76a27e9" path="/var/lib/kubelet/pods/3d53b1de-13a4-4639-8cbc-e8f2d76a27e9/volumes"
Sep 15 21:56:02.750727 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:02.750708    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="def6dd16-fb1a-4d0f-9881-51d147f35833" path="/var/lib/kubelet/pods/def6dd16-fb1a-4d0f-9881-51d147f35833/volumes"
Sep 15 21:56:03.861872 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:03.861838    2575 scope.go:117] "RemoveContainer" containerID="176cb69b0734cc3644df70258107fa06477543c7f9861faefd1dae7c8bda67c1"
Sep 15 21:56:04.843611 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:04.843551    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m" podUID="a0a857bb-1788-4de0-b3cf-0bebbb5b1243" containerName="main" containerID="cri-o://1bdbaf275ae912b0c75cf2c349ab350e6eb5efb965241d951ffde9052e7c2578" gracePeriod=45
Sep 15 21:56:04.865207 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:04.865101    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5" podUID="5e26b766-dc93-450d-abf6-676197093a17" containerName="main" containerID="cri-o://ca4128fcc61957c22af9d739376b62caab7cdb8af27c99fee984093a3135199c" gracePeriod=45
Sep 15 21:56:04.867681 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:04.867648    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d" podUID="bb08497b-950e-4db1-ac06-0ba91361da32" containerName="main" containerID="cri-o://330c208356462a6419547833ce82348c17a196600a47eeb0a488efd40e81dc3e" gracePeriod=45
Sep 15 21:56:05.117397 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.117347    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:56:05.176187 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.176160    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:56:05.178239 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.178219    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:56:05.234701 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.234672    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-tls-certs\") pod \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") "
Sep 15 21:56:05.234908 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.234721    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zd9j\" (UniqueName: \"kubernetes.io/projected/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-kube-api-access-9zd9j\") pod \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") "
Sep 15 21:56:05.234908 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.234787    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6wt4\" (UniqueName: \"kubernetes.io/projected/5e26b766-dc93-450d-abf6-676197093a17-kube-api-access-z6wt4\") pod \"5e26b766-dc93-450d-abf6-676197093a17\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") "
Sep 15 21:56:05.234908 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.234822    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-dshm\") pod \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") "
Sep 15 21:56:05.234908 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.234869    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-model-cache\") pod \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") "
Sep 15 21:56:05.234908 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.234900    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-tmp-dir\") pod \"5e26b766-dc93-450d-abf6-676197093a17\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") "
Sep 15 21:56:05.235176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.234944    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-home\") pod \"bb08497b-950e-4db1-ac06-0ba91361da32\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") "
Sep 15 21:56:05.235176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.234982    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-model-cache\") pod \"5e26b766-dc93-450d-abf6-676197093a17\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") "
Sep 15 21:56:05.235176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.235005    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-home\") pod \"5e26b766-dc93-450d-abf6-676197093a17\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") "
Sep 15 21:56:05.235176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.235041    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfhjt\" (UniqueName: \"kubernetes.io/projected/bb08497b-950e-4db1-ac06-0ba91361da32-kube-api-access-nfhjt\") pod \"bb08497b-950e-4db1-ac06-0ba91361da32\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") "
Sep 15 21:56:05.235176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.235069    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-dshm\") pod \"bb08497b-950e-4db1-ac06-0ba91361da32\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") "
Sep 15 21:56:05.235176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.235091    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-dshm\") pod \"5e26b766-dc93-450d-abf6-676197093a17\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") "
Sep 15 21:56:05.235176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.235128    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-home\") pod \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") "
Sep 15 21:56:05.235579 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.235208    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-tmp-dir\") pod \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\" (UID: \"a0a857bb-1788-4de0-b3cf-0bebbb5b1243\") "
Sep 15 21:56:05.235579 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.235234    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-model-cache\") pod \"bb08497b-950e-4db1-ac06-0ba91361da32\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") "
Sep 15 21:56:05.235579 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.235252    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0a857bb-1788-4de0-b3cf-0bebbb5b1243" (UID: "a0a857bb-1788-4de0-b3cf-0bebbb5b1243"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:56:05.235579 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.235263    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-tmp-dir\") pod \"bb08497b-950e-4db1-ac06-0ba91361da32\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") "
Sep 15 21:56:05.235579 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.235480    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb08497b-950e-4db1-ac06-0ba91361da32" (UID: "bb08497b-950e-4db1-ac06-0ba91361da32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:56:05.235877 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.235740    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-home" (OuterVolumeSpecName: "home") pod "a0a857bb-1788-4de0-b3cf-0bebbb5b1243" (UID: "a0a857bb-1788-4de0-b3cf-0bebbb5b1243"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:56:05.236248 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.235975    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e26b766-dc93-450d-abf6-676197093a17" (UID: "5e26b766-dc93-450d-abf6-676197093a17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:56:05.236248 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.236056    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e26b766-dc93-450d-abf6-676197093a17" (UID: "5e26b766-dc93-450d-abf6-676197093a17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:56:05.236248 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.236178    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-home" (OuterVolumeSpecName: "home") pod "bb08497b-950e-4db1-ac06-0ba91361da32" (UID: "bb08497b-950e-4db1-ac06-0ba91361da32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:56:05.236567 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.236262    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0a857bb-1788-4de0-b3cf-0bebbb5b1243" (UID: "a0a857bb-1788-4de0-b3cf-0bebbb5b1243"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:56:05.236567 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.236322    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e26b766-dc93-450d-abf6-676197093a17-tls-certs\") pod \"5e26b766-dc93-450d-abf6-676197093a17\" (UID: \"5e26b766-dc93-450d-abf6-676197093a17\") "
Sep 15 21:56:05.236567 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.236356    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb08497b-950e-4db1-ac06-0ba91361da32-tls-certs\") pod \"bb08497b-950e-4db1-ac06-0ba91361da32\" (UID: \"bb08497b-950e-4db1-ac06-0ba91361da32\") "
Sep 15 21:56:05.236567 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.236568    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-home" (OuterVolumeSpecName: "home") pod "5e26b766-dc93-450d-abf6-676197093a17" (UID: "5e26b766-dc93-450d-abf6-676197093a17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:56:05.236567 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.236578    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb08497b-950e-4db1-ac06-0ba91361da32" (UID: "bb08497b-950e-4db1-ac06-0ba91361da32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:56:05.236955 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.236879    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.236955 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.236902    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.236955 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.236916    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.236955 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.236928    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.236955 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.236942    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.236955 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.236956    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.237301 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.236969    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.237301 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.236982    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.237301 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.236994    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.237590 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.237545    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e26b766-dc93-450d-abf6-676197093a17-kube-api-access-z6wt4" (OuterVolumeSpecName: "kube-api-access-z6wt4") pod "5e26b766-dc93-450d-abf6-676197093a17" (UID: "5e26b766-dc93-450d-abf6-676197093a17"). InnerVolumeSpecName "kube-api-access-z6wt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:56:05.239110 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.239051    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-dshm" (OuterVolumeSpecName: "dshm") pod "bb08497b-950e-4db1-ac06-0ba91361da32" (UID: "bb08497b-950e-4db1-ac06-0ba91361da32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:56:05.239230 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.239183    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-kube-api-access-9zd9j" (OuterVolumeSpecName: "kube-api-access-9zd9j") pod "a0a857bb-1788-4de0-b3cf-0bebbb5b1243" (UID: "a0a857bb-1788-4de0-b3cf-0bebbb5b1243"). InnerVolumeSpecName "kube-api-access-9zd9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:56:05.239556 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.239530    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0a857bb-1788-4de0-b3cf-0bebbb5b1243" (UID: "a0a857bb-1788-4de0-b3cf-0bebbb5b1243"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:56:05.239770 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.239720    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-dshm" (OuterVolumeSpecName: "dshm") pod "5e26b766-dc93-450d-abf6-676197093a17" (UID: "5e26b766-dc93-450d-abf6-676197093a17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:56:05.239868 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.239792    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb08497b-950e-4db1-ac06-0ba91361da32-kube-api-access-nfhjt" (OuterVolumeSpecName: "kube-api-access-nfhjt") pod "bb08497b-950e-4db1-ac06-0ba91361da32" (UID: "bb08497b-950e-4db1-ac06-0ba91361da32"). InnerVolumeSpecName "kube-api-access-nfhjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:56:05.239961 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.239942    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-dshm" (OuterVolumeSpecName: "dshm") pod "a0a857bb-1788-4de0-b3cf-0bebbb5b1243" (UID: "a0a857bb-1788-4de0-b3cf-0bebbb5b1243"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:56:05.240176 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.240159    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e26b766-dc93-450d-abf6-676197093a17-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e26b766-dc93-450d-abf6-676197093a17" (UID: "5e26b766-dc93-450d-abf6-676197093a17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:56:05.240709 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.240694    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb08497b-950e-4db1-ac06-0ba91361da32-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb08497b-950e-4db1-ac06-0ba91361da32" (UID: "bb08497b-950e-4db1-ac06-0ba91361da32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:56:05.337638 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.337603    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e26b766-dc93-450d-abf6-676197093a17-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.337638 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.337630    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb08497b-950e-4db1-ac06-0ba91361da32-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.337638 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.337639    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.337903 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.337648    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zd9j\" (UniqueName: \"kubernetes.io/projected/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-kube-api-access-9zd9j\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.337903 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.337658    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6wt4\" (UniqueName: \"kubernetes.io/projected/5e26b766-dc93-450d-abf6-676197093a17-kube-api-access-z6wt4\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.337903 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.337675    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0a857bb-1788-4de0-b3cf-0bebbb5b1243-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.337903 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.337684    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfhjt\" (UniqueName: \"kubernetes.io/projected/bb08497b-950e-4db1-ac06-0ba91361da32-kube-api-access-nfhjt\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.337903 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.337693    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb08497b-950e-4db1-ac06-0ba91361da32-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.337903 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.337701    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e26b766-dc93-450d-abf6-676197093a17-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:56:05.869398 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.869363    2575 generic.go:356] "Generic (PLEG): container finished" podID="5e26b766-dc93-450d-abf6-676197093a17" containerID="ca4128fcc61957c22af9d739376b62caab7cdb8af27c99fee984093a3135199c" exitCode=0
Sep 15 21:56:05.869783 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.869408    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"
Sep 15 21:56:05.869783 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.869455    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5" event={"ID":"5e26b766-dc93-450d-abf6-676197093a17","Type":"ContainerDied","Data":"ca4128fcc61957c22af9d739376b62caab7cdb8af27c99fee984093a3135199c"}
Sep 15 21:56:05.869783 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.869500    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5" event={"ID":"5e26b766-dc93-450d-abf6-676197093a17","Type":"ContainerDied","Data":"2de140ce8095833ce750182fd6adafe9219d7e2a9651540776a6373088a908f1"}
Sep 15 21:56:05.869783 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.869522    2575 scope.go:117] "RemoveContainer" containerID="ca4128fcc61957c22af9d739376b62caab7cdb8af27c99fee984093a3135199c"
Sep 15 21:56:05.870954 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.870817    2575 generic.go:356] "Generic (PLEG): container finished" podID="bb08497b-950e-4db1-ac06-0ba91361da32" containerID="330c208356462a6419547833ce82348c17a196600a47eeb0a488efd40e81dc3e" exitCode=0
Sep 15 21:56:05.870954 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.870877    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"
Sep 15 21:56:05.870954 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.870898    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d" event={"ID":"bb08497b-950e-4db1-ac06-0ba91361da32","Type":"ContainerDied","Data":"330c208356462a6419547833ce82348c17a196600a47eeb0a488efd40e81dc3e"}
Sep 15 21:56:05.870954 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.870929    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d" event={"ID":"bb08497b-950e-4db1-ac06-0ba91361da32","Type":"ContainerDied","Data":"118df60534f4d9810c0c8eb75db2449d27c0b47359e8d487aaad2d3d0c3782fa"}
Sep 15 21:56:05.872379 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.872356    2575 generic.go:356] "Generic (PLEG): container finished" podID="a0a857bb-1788-4de0-b3cf-0bebbb5b1243" containerID="1bdbaf275ae912b0c75cf2c349ab350e6eb5efb965241d951ffde9052e7c2578" exitCode=0
Sep 15 21:56:05.872500 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.872388    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"
Sep 15 21:56:05.872500 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.872404    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m" event={"ID":"a0a857bb-1788-4de0-b3cf-0bebbb5b1243","Type":"ContainerDied","Data":"1bdbaf275ae912b0c75cf2c349ab350e6eb5efb965241d951ffde9052e7c2578"}
Sep 15 21:56:05.872500 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.872432    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m" event={"ID":"a0a857bb-1788-4de0-b3cf-0bebbb5b1243","Type":"ContainerDied","Data":"bade718401b54c67b8da774855fa41eb16488728fee6ef46c8baedccb8ac23f1"}
Sep 15 21:56:05.879835 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.879822    2575 scope.go:117] "RemoveContainer" containerID="ca4128fcc61957c22af9d739376b62caab7cdb8af27c99fee984093a3135199c"
Sep 15 21:56:05.880116 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:56:05.880098    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca4128fcc61957c22af9d739376b62caab7cdb8af27c99fee984093a3135199c\": container with ID starting with ca4128fcc61957c22af9d739376b62caab7cdb8af27c99fee984093a3135199c not found: ID does not exist" containerID="ca4128fcc61957c22af9d739376b62caab7cdb8af27c99fee984093a3135199c"
Sep 15 21:56:05.880180 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.880125    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca4128fcc61957c22af9d739376b62caab7cdb8af27c99fee984093a3135199c"} err="failed to get container status \"ca4128fcc61957c22af9d739376b62caab7cdb8af27c99fee984093a3135199c\": rpc error: code = NotFound desc = could not find container \"ca4128fcc61957c22af9d739376b62caab7cdb8af27c99fee984093a3135199c\": container with ID starting with ca4128fcc61957c22af9d739376b62caab7cdb8af27c99fee984093a3135199c not found: ID does not exist"
Sep 15 21:56:05.880180 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.880140    2575 scope.go:117] "RemoveContainer" containerID="330c208356462a6419547833ce82348c17a196600a47eeb0a488efd40e81dc3e"
Sep 15 21:56:05.889046 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.889030    2575 scope.go:117] "RemoveContainer" containerID="330c208356462a6419547833ce82348c17a196600a47eeb0a488efd40e81dc3e"
Sep 15 21:56:05.889314 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:56:05.889298    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330c208356462a6419547833ce82348c17a196600a47eeb0a488efd40e81dc3e\": container with ID starting with 330c208356462a6419547833ce82348c17a196600a47eeb0a488efd40e81dc3e not found: ID does not exist" containerID="330c208356462a6419547833ce82348c17a196600a47eeb0a488efd40e81dc3e"
Sep 15 21:56:05.889353 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.889323    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330c208356462a6419547833ce82348c17a196600a47eeb0a488efd40e81dc3e"} err="failed to get container status \"330c208356462a6419547833ce82348c17a196600a47eeb0a488efd40e81dc3e\": rpc error: code = NotFound desc = could not find container \"330c208356462a6419547833ce82348c17a196600a47eeb0a488efd40e81dc3e\": container with ID starting with 330c208356462a6419547833ce82348c17a196600a47eeb0a488efd40e81dc3e not found: ID does not exist"
Sep 15 21:56:05.889353 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.889338    2575 scope.go:117] "RemoveContainer" containerID="1bdbaf275ae912b0c75cf2c349ab350e6eb5efb965241d951ffde9052e7c2578"
Sep 15 21:56:05.897118 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.897099    2575 scope.go:117] "RemoveContainer" containerID="1bdbaf275ae912b0c75cf2c349ab350e6eb5efb965241d951ffde9052e7c2578"
Sep 15 21:56:05.897362 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:56:05.897345    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bdbaf275ae912b0c75cf2c349ab350e6eb5efb965241d951ffde9052e7c2578\": container with ID starting with 1bdbaf275ae912b0c75cf2c349ab350e6eb5efb965241d951ffde9052e7c2578 not found: ID does not exist" containerID="1bdbaf275ae912b0c75cf2c349ab350e6eb5efb965241d951ffde9052e7c2578"
Sep 15 21:56:05.897424 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.897373    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bdbaf275ae912b0c75cf2c349ab350e6eb5efb965241d951ffde9052e7c2578"} err="failed to get container status \"1bdbaf275ae912b0c75cf2c349ab350e6eb5efb965241d951ffde9052e7c2578\": rpc error: code = NotFound desc = could not find container \"1bdbaf275ae912b0c75cf2c349ab350e6eb5efb965241d951ffde9052e7c2578\": container with ID starting with 1bdbaf275ae912b0c75cf2c349ab350e6eb5efb965241d951ffde9052e7c2578 not found: ID does not exist"
Sep 15 21:56:05.905387 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.905363    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"]
Sep 15 21:56:05.909273 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.909255    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/distinct-w4-56e305-kserve-5854658cd7-478k5"]
Sep 15 21:56:05.936811 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.936778    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"]
Sep 15 21:56:05.940164 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.940145    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/simulator-w4-56e305-kserve-798766b64f-z8g4d"]
Sep 15 21:56:05.955115 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.955092    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"]
Sep 15 21:56:05.967334 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:05.967311    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-56e305/distinct-2-w4-56e305-kserve-749cd8f7b4-j5j4m"]
Sep 15 21:56:06.748484 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:06.748449    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e26b766-dc93-450d-abf6-676197093a17" path="/var/lib/kubelet/pods/5e26b766-dc93-450d-abf6-676197093a17/volumes"
Sep 15 21:56:06.748871 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:06.748857    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0a857bb-1788-4de0-b3cf-0bebbb5b1243" path="/var/lib/kubelet/pods/a0a857bb-1788-4de0-b3cf-0bebbb5b1243/volumes"
Sep 15 21:56:06.749204 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:06.749193    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb08497b-950e-4db1-ac06-0ba91361da32" path="/var/lib/kubelet/pods/bb08497b-950e-4db1-ac06-0ba91361da32/volumes"
Sep 15 21:56:11.251876 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:11.251835    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:56:49.779491 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.779454    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"]
Sep 15 21:56:49.779921 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.779873    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d53b1de-13a4-4639-8cbc-e8f2d76a27e9" containerName="revoke-keys"
Sep 15 21:56:49.779921 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.779885    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d53b1de-13a4-4639-8cbc-e8f2d76a27e9" containerName="revoke-keys"
Sep 15 21:56:49.779921 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.779899    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="def6dd16-fb1a-4d0f-9881-51d147f35833" containerName="main"
Sep 15 21:56:49.779921 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.779904    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="def6dd16-fb1a-4d0f-9881-51d147f35833" containerName="main"
Sep 15 21:56:49.779921 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.779917    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb08497b-950e-4db1-ac06-0ba91361da32" containerName="main"
Sep 15 21:56:49.779921 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.779922    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb08497b-950e-4db1-ac06-0ba91361da32" containerName="main"
Sep 15 21:56:49.780117 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.779938    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e26b766-dc93-450d-abf6-676197093a17" containerName="main"
Sep 15 21:56:49.780117 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.779944    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e26b766-dc93-450d-abf6-676197093a17" containerName="main"
Sep 15 21:56:49.780117 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.779952    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a857bb-1788-4de0-b3cf-0bebbb5b1243" containerName="main"
Sep 15 21:56:49.780117 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.779959    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a857bb-1788-4de0-b3cf-0bebbb5b1243" containerName="main"
Sep 15 21:56:49.780117 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.780019    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e26b766-dc93-450d-abf6-676197093a17" containerName="main"
Sep 15 21:56:49.780117 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.780026    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a857bb-1788-4de0-b3cf-0bebbb5b1243" containerName="main"
Sep 15 21:56:49.780117 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.780032    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="def6dd16-fb1a-4d0f-9881-51d147f35833" containerName="main"
Sep 15 21:56:49.780117 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.780040    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb08497b-950e-4db1-ac06-0ba91361da32" containerName="main"
Sep 15 21:56:49.780117 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.780046    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d53b1de-13a4-4639-8cbc-e8f2d76a27e9" containerName="revoke-keys"
Sep 15 21:56:49.783260 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.783239    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:49.788925 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.788903    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e7a648\"/\"openshift-service-ca.crt\""
Sep 15 21:56:49.790016 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.789777    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e7a648\"/\"default-dockercfg-vxmvf\""
Sep 15 21:56:49.790016 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.789815    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e7a648\"/\"kube-root-ca.crt\""
Sep 15 21:56:49.790016 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.790003    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e7a648\"/\"sub-test-model-e7a648-kserve-self-signed-certs\""
Sep 15 21:56:49.807005 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.806974    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"]
Sep 15 21:56:49.943149 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.943113    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-dshm\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:49.943149 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.943158    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-model-cache\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:49.943387 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.943177    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-tmp-dir\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:49.943387 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.943270    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7l6z\" (UniqueName: \"kubernetes.io/projected/f21ca1ef-4900-43d9-b3de-07937856363b-kube-api-access-x7l6z\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:49.943387 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.943343    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-home\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:49.943387 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:49.943363    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f21ca1ef-4900-43d9-b3de-07937856363b-tls-certs\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:50.044358 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:50.044237    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-home\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:50.044358 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:50.044295    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f21ca1ef-4900-43d9-b3de-07937856363b-tls-certs\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:50.044358 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:50.044348    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-dshm\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:50.044667 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:50.044373    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-model-cache\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:50.044667 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:50.044396    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-tmp-dir\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:50.044667 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:50.044445    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7l6z\" (UniqueName: \"kubernetes.io/projected/f21ca1ef-4900-43d9-b3de-07937856363b-kube-api-access-x7l6z\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:50.044883 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:50.044707    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-home\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:50.044883 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:50.044803    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-model-cache\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:50.044883 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:50.044868    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-tmp-dir\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:50.046920 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:50.046900    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-dshm\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:50.047011 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:50.046995    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f21ca1ef-4900-43d9-b3de-07937856363b-tls-certs\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:50.056152 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:50.056125    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7l6z\" (UniqueName: \"kubernetes.io/projected/f21ca1ef-4900-43d9-b3de-07937856363b-kube-api-access-x7l6z\") pod \"sub-test-model-e7a648-kserve-9b7c66c89-fcjl2\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") " pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:50.092306 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:50.092258    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:56:50.382257 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:50.382123    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"]
Sep 15 21:56:50.385009 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:56:50.384983    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf21ca1ef_4900_43d9_b3de_07937856363b.slice/crio-6d05cbcb4cad4323b99fb3df128c7a3d8de9984f33d378b56491c375fb505a5c WatchSource:0}: Error finding container 6d05cbcb4cad4323b99fb3df128c7a3d8de9984f33d378b56491c375fb505a5c: Status 404 returned error can't find the container with id 6d05cbcb4cad4323b99fb3df128c7a3d8de9984f33d378b56491c375fb505a5c
Sep 15 21:56:51.106254 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:51.106220    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2" event={"ID":"f21ca1ef-4900-43d9-b3de-07937856363b","Type":"ContainerStarted","Data":"76ceb02234c5abae320686319fb847a39bd4ddf6bec3e97f282450c741a200d9"}
Sep 15 21:56:51.106636 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:51.106267    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2" event={"ID":"f21ca1ef-4900-43d9-b3de-07937856363b","Type":"ContainerStarted","Data":"6d05cbcb4cad4323b99fb3df128c7a3d8de9984f33d378b56491c375fb505a5c"}
Sep 15 21:56:53.492868 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:53.492815    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2" podStartSLOduration=4.492797419 podStartE2EDuration="4.492797419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:56:51.146402718 +0000 UTC m=+1848.958411161" watchObservedRunningTime="2026-09-15 21:56:53.492797419 +0000 UTC m=+1851.304805881"
Sep 15 21:56:53.494365 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:56:53.494345    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:57:00.093345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:00.093300    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:57:00.093345 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:00.093350    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:57:00.105628 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:00.105606    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:57:00.165706 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:00.165679    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:57:20.571044 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:20.571009    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:57:31.291484 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:31.291405    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7728-092256"]
Sep 15 21:57:31.295285 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:31.295268    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7728-092256"
Sep 15 21:57:31.299640 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:31.299620    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2lhf2\""
Sep 15 21:57:31.307552 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:31.307524    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7728-092256"]
Sep 15 21:57:31.444162 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:31.444124    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2d8s\" (UniqueName: \"kubernetes.io/projected/6eb77855-788c-4ae7-96a1-43f4cb3d2be8-kube-api-access-p2d8s\") pod \"test-curl-7728-092256\" (UID: \"6eb77855-788c-4ae7-96a1-43f4cb3d2be8\") " pod="opendatahub/test-curl-7728-092256"
Sep 15 21:57:31.545578 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:31.545487    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2d8s\" (UniqueName: \"kubernetes.io/projected/6eb77855-788c-4ae7-96a1-43f4cb3d2be8-kube-api-access-p2d8s\") pod \"test-curl-7728-092256\" (UID: \"6eb77855-788c-4ae7-96a1-43f4cb3d2be8\") " pod="opendatahub/test-curl-7728-092256"
Sep 15 21:57:31.561991 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:31.561957    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2d8s\" (UniqueName: \"kubernetes.io/projected/6eb77855-788c-4ae7-96a1-43f4cb3d2be8-kube-api-access-p2d8s\") pod \"test-curl-7728-092256\" (UID: \"6eb77855-788c-4ae7-96a1-43f4cb3d2be8\") " pod="opendatahub/test-curl-7728-092256"
Sep 15 21:57:31.603981 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:31.603943    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7728-092256"
Sep 15 21:57:31.888613 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:31.888580    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7728-092256"]
Sep 15 21:57:31.892190 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:57:31.892163    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6eb77855_788c_4ae7_96a1_43f4cb3d2be8.slice/crio-e9bd988a8b8138ffddd7f809979e27c3728bd2a62873f936ccae75ce2037fa5c WatchSource:0}: Error finding container e9bd988a8b8138ffddd7f809979e27c3728bd2a62873f936ccae75ce2037fa5c: Status 404 returned error can't find the container with id e9bd988a8b8138ffddd7f809979e27c3728bd2a62873f936ccae75ce2037fa5c
Sep 15 21:57:32.321090 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:32.321004    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7728-092256" event={"ID":"6eb77855-788c-4ae7-96a1-43f4cb3d2be8","Type":"ContainerStarted","Data":"f7f68b597a6e256cc23f8bc163651fa437b2680e4d6a1991de68b145ff41b81e"}
Sep 15 21:57:32.321090 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:32.321053    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7728-092256" event={"ID":"6eb77855-788c-4ae7-96a1-43f4cb3d2be8","Type":"ContainerStarted","Data":"e9bd988a8b8138ffddd7f809979e27c3728bd2a62873f936ccae75ce2037fa5c"}
Sep 15 21:57:32.360604 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:32.360550    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7728-092256" podStartSLOduration=1.360532904 podStartE2EDuration="1.360532904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:57:32.359054931 +0000 UTC m=+1890.171063375" watchObservedRunningTime="2026-09-15 21:57:32.360532904 +0000 UTC m=+1890.172541402"
Sep 15 21:57:33.200507 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:33.200421    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7728-092256"]
Sep 15 21:57:33.216271 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:33.216240    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7728-092256"]
Sep 15 21:57:33.326209 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:57:33.326180    2575 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7728-092256" container="test-curl-7728-092256"
Sep 15 21:57:33.345830 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:33.345799    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7728-3d0391"]
Sep 15 21:57:33.346198 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:33.346186    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eb77855-788c-4ae7-96a1-43f4cb3d2be8" containerName="test-curl-7728-092256"
Sep 15 21:57:33.346241 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:33.346200    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eb77855-788c-4ae7-96a1-43f4cb3d2be8" containerName="test-curl-7728-092256"
Sep 15 21:57:33.346276 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:33.346263    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eb77855-788c-4ae7-96a1-43f4cb3d2be8" containerName="test-curl-7728-092256"
Sep 15 21:57:33.349311 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:33.349296    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7728-3d0391"
Sep 15 21:57:33.358364 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:33.358339    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7728-3d0391"]
Sep 15 21:57:33.464676 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:33.464573    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztl99\" (UniqueName: \"kubernetes.io/projected/ab32b9a9-5c90-4360-9491-4fa6bad782c9-kube-api-access-ztl99\") pod \"test-curl-7728-3d0391\" (UID: \"ab32b9a9-5c90-4360-9491-4fa6bad782c9\") " pod="opendatahub/test-curl-7728-3d0391"
Sep 15 21:57:33.565113 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:33.565077    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztl99\" (UniqueName: \"kubernetes.io/projected/ab32b9a9-5c90-4360-9491-4fa6bad782c9-kube-api-access-ztl99\") pod \"test-curl-7728-3d0391\" (UID: \"ab32b9a9-5c90-4360-9491-4fa6bad782c9\") " pod="opendatahub/test-curl-7728-3d0391"
Sep 15 21:57:33.582795 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:33.582769    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztl99\" (UniqueName: \"kubernetes.io/projected/ab32b9a9-5c90-4360-9491-4fa6bad782c9-kube-api-access-ztl99\") pod \"test-curl-7728-3d0391\" (UID: \"ab32b9a9-5c90-4360-9491-4fa6bad782c9\") " pod="opendatahub/test-curl-7728-3d0391"
Sep 15 21:57:33.659336 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:33.659301    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7728-3d0391"
Sep 15 21:57:33.951623 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:33.951596    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7728-3d0391"]
Sep 15 21:57:33.953846 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:57:33.953820    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab32b9a9_5c90_4360_9491_4fa6bad782c9.slice/crio-5c6c040f247c611d12152a4e1dec197e6e092f84ce213630f9432e0bb8fd0856 WatchSource:0}: Error finding container 5c6c040f247c611d12152a4e1dec197e6e092f84ce213630f9432e0bb8fd0856: Status 404 returned error can't find the container with id 5c6c040f247c611d12152a4e1dec197e6e092f84ce213630f9432e0bb8fd0856
Sep 15 21:57:34.332397 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:34.332298    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7728-3d0391" event={"ID":"ab32b9a9-5c90-4360-9491-4fa6bad782c9","Type":"ContainerStarted","Data":"872be22603bd23504a38211cb467817b878419d7a348c27176190207449ca538"}
Sep 15 21:57:34.332397 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:34.332347    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7728-3d0391" event={"ID":"ab32b9a9-5c90-4360-9491-4fa6bad782c9","Type":"ContainerStarted","Data":"5c6c040f247c611d12152a4e1dec197e6e092f84ce213630f9432e0bb8fd0856"}
Sep 15 21:57:34.332397 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:34.332376    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7728-092256" podUID="6eb77855-788c-4ae7-96a1-43f4cb3d2be8" containerName="test-curl-7728-092256" containerID="cri-o://f7f68b597a6e256cc23f8bc163651fa437b2680e4d6a1991de68b145ff41b81e" gracePeriod=2
Sep 15 21:57:34.357115 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:34.357069    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7728-3d0391" podStartSLOduration=1.357053058 podStartE2EDuration="1.357053058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:57:34.354955189 +0000 UTC m=+1892.166963633" watchObservedRunningTime="2026-09-15 21:57:34.357053058 +0000 UTC m=+1892.169061501"
Sep 15 21:57:34.357347 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:57:34.357317    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7728-092256\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="6eb77855-788c-4ae7-96a1-43f4cb3d2be8" pod="opendatahub/test-curl-7728-092256"
Sep 15 21:57:34.359308 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:57:34.359287    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7728-092256\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="6eb77855-788c-4ae7-96a1-43f4cb3d2be8" pod="opendatahub/test-curl-7728-092256"
Sep 15 21:57:35.187123 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:35.187038    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7728-3d0391"]
Sep 15 21:57:35.192311 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:35.192279    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7728-3d0391"]
Sep 15 21:57:35.338050 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:57:35.338016    2575 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7728-3d0391" container="test-curl-7728-3d0391"
Sep 15 21:57:36.343863 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:36.343825    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7728-3d0391" podUID="ab32b9a9-5c90-4360-9491-4fa6bad782c9" containerName="test-curl-7728-3d0391" containerID="cri-o://872be22603bd23504a38211cb467817b878419d7a348c27176190207449ca538" gracePeriod=2
Sep 15 21:57:36.347207 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:57:36.347183    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7728-3d0391\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="ab32b9a9-5c90-4360-9491-4fa6bad782c9" pod="opendatahub/test-curl-7728-3d0391"
Sep 15 21:57:36.525910 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:36.525886    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7728-092256"
Sep 15 21:57:36.530086 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:57:36.530062    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7728-3d0391\" is forbidden: User \"system:node:ip-10-0-135-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-148.ec2.internal' and this object" podUID="ab32b9a9-5c90-4360-9491-4fa6bad782c9" pod="opendatahub/test-curl-7728-3d0391"
Sep 15 21:57:36.577959 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:36.577920    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:57:36.694695 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:36.694659    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2d8s\" (UniqueName: \"kubernetes.io/projected/6eb77855-788c-4ae7-96a1-43f4cb3d2be8-kube-api-access-p2d8s\") pod \"6eb77855-788c-4ae7-96a1-43f4cb3d2be8\" (UID: \"6eb77855-788c-4ae7-96a1-43f4cb3d2be8\") "
Sep 15 21:57:36.696963 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:36.696937    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eb77855-788c-4ae7-96a1-43f4cb3d2be8-kube-api-access-p2d8s" (OuterVolumeSpecName: "kube-api-access-p2d8s") pod "6eb77855-788c-4ae7-96a1-43f4cb3d2be8" (UID: "6eb77855-788c-4ae7-96a1-43f4cb3d2be8"). InnerVolumeSpecName "kube-api-access-p2d8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:57:36.747014 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:36.746982    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eb77855-788c-4ae7-96a1-43f4cb3d2be8" path="/var/lib/kubelet/pods/6eb77855-788c-4ae7-96a1-43f4cb3d2be8/volumes"
Sep 15 21:57:36.795695 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:36.795660    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2d8s\" (UniqueName: \"kubernetes.io/projected/6eb77855-788c-4ae7-96a1-43f4cb3d2be8-kube-api-access-p2d8s\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:57:37.350196 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:37.350162    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7728-092256"
Sep 15 21:57:37.350196 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:37.350200    2575 scope.go:117] "RemoveContainer" containerID="f7f68b597a6e256cc23f8bc163651fa437b2680e4d6a1991de68b145ff41b81e"
Sep 15 21:57:37.350718 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:37.350163    2575 generic.go:356] "Generic (PLEG): container finished" podID="6eb77855-788c-4ae7-96a1-43f4cb3d2be8" containerID="f7f68b597a6e256cc23f8bc163651fa437b2680e4d6a1991de68b145ff41b81e" exitCode=137
Sep 15 21:57:37.359545 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:37.359527    2575 scope.go:117] "RemoveContainer" containerID="f7f68b597a6e256cc23f8bc163651fa437b2680e4d6a1991de68b145ff41b81e"
Sep 15 21:57:37.359854 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:57:37.359835    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f68b597a6e256cc23f8bc163651fa437b2680e4d6a1991de68b145ff41b81e\": container with ID starting with f7f68b597a6e256cc23f8bc163651fa437b2680e4d6a1991de68b145ff41b81e not found: ID does not exist" containerID="f7f68b597a6e256cc23f8bc163651fa437b2680e4d6a1991de68b145ff41b81e"
Sep 15 21:57:37.359909 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:37.359865    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f68b597a6e256cc23f8bc163651fa437b2680e4d6a1991de68b145ff41b81e"} err="failed to get container status \"f7f68b597a6e256cc23f8bc163651fa437b2680e4d6a1991de68b145ff41b81e\": rpc error: code = NotFound desc = could not find container \"f7f68b597a6e256cc23f8bc163651fa437b2680e4d6a1991de68b145ff41b81e\": container with ID starting with f7f68b597a6e256cc23f8bc163651fa437b2680e4d6a1991de68b145ff41b81e not found: ID does not exist"
Sep 15 21:57:38.473683 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:38.473645    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"]
Sep 15 21:57:38.551295 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:38.551272    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7728-3d0391"
Sep 15 21:57:38.716955 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:38.716864    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztl99\" (UniqueName: \"kubernetes.io/projected/ab32b9a9-5c90-4360-9491-4fa6bad782c9-kube-api-access-ztl99\") pod \"ab32b9a9-5c90-4360-9491-4fa6bad782c9\" (UID: \"ab32b9a9-5c90-4360-9491-4fa6bad782c9\") "
Sep 15 21:57:38.719114 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:38.719090    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab32b9a9-5c90-4360-9491-4fa6bad782c9-kube-api-access-ztl99" (OuterVolumeSpecName: "kube-api-access-ztl99") pod "ab32b9a9-5c90-4360-9491-4fa6bad782c9" (UID: "ab32b9a9-5c90-4360-9491-4fa6bad782c9"). InnerVolumeSpecName "kube-api-access-ztl99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:57:38.747479 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:38.747450    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab32b9a9-5c90-4360-9491-4fa6bad782c9" path="/var/lib/kubelet/pods/ab32b9a9-5c90-4360-9491-4fa6bad782c9/volumes"
Sep 15 21:57:38.818182 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:38.818152    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztl99\" (UniqueName: \"kubernetes.io/projected/ab32b9a9-5c90-4360-9491-4fa6bad782c9-kube-api-access-ztl99\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:57:39.362525 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:39.362484    2575 generic.go:356] "Generic (PLEG): container finished" podID="ab32b9a9-5c90-4360-9491-4fa6bad782c9" containerID="872be22603bd23504a38211cb467817b878419d7a348c27176190207449ca538" exitCode=137
Sep 15 21:57:39.362525 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:39.362529    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7728-3d0391"
Sep 15 21:57:39.362801 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:39.362577    2575 scope.go:117] "RemoveContainer" containerID="872be22603bd23504a38211cb467817b878419d7a348c27176190207449ca538"
Sep 15 21:57:39.371126 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:39.371104    2575 scope.go:117] "RemoveContainer" containerID="872be22603bd23504a38211cb467817b878419d7a348c27176190207449ca538"
Sep 15 21:57:39.371383 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:57:39.371367    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"872be22603bd23504a38211cb467817b878419d7a348c27176190207449ca538\": container with ID starting with 872be22603bd23504a38211cb467817b878419d7a348c27176190207449ca538 not found: ID does not exist" containerID="872be22603bd23504a38211cb467817b878419d7a348c27176190207449ca538"
Sep 15 21:57:39.371435 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:39.371389    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"872be22603bd23504a38211cb467817b878419d7a348c27176190207449ca538"} err="failed to get container status \"872be22603bd23504a38211cb467817b878419d7a348c27176190207449ca538\": rpc error: code = NotFound desc = could not find container \"872be22603bd23504a38211cb467817b878419d7a348c27176190207449ca538\": container with ID starting with 872be22603bd23504a38211cb467817b878419d7a348c27176190207449ca538 not found: ID does not exist"
Sep 15 21:57:47.160280 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:47.160243    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:57:48.276542 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.276502    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"]
Sep 15 21:57:48.276941 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.276927    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab32b9a9-5c90-4360-9491-4fa6bad782c9" containerName="test-curl-7728-3d0391"
Sep 15 21:57:48.276941 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.276942    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab32b9a9-5c90-4360-9491-4fa6bad782c9" containerName="test-curl-7728-3d0391"
Sep 15 21:57:48.277018 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.277013    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab32b9a9-5c90-4360-9491-4fa6bad782c9" containerName="test-curl-7728-3d0391"
Sep 15 21:57:48.281431 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.281415    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.285989 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.285964    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce3ab0\"/\"default-dockercfg-lxhpx\""
Sep 15 21:57:48.286145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.285964    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce3ab0\"/\"openshift-service-ca.crt\""
Sep 15 21:57:48.286145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.285966    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce3ab0\"/\"rate-test-model-ce3ab0-kserve-self-signed-certs\""
Sep 15 21:57:48.286145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.285964    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ce3ab0\"/\"kube-root-ca.crt\""
Sep 15 21:57:48.300969 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.300943    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/013dad5f-f574-48fc-a206-f30aac8de31b-tls-certs\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.301110 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.300979    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-model-cache\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.301110 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.301048    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrj65\" (UniqueName: \"kubernetes.io/projected/013dad5f-f574-48fc-a206-f30aac8de31b-kube-api-access-mrj65\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.301110 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.301090    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-dshm\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.301260 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.301118    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-home\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.301260 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.301216    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-tmp-dir\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.309879 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.306799    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"]
Sep 15 21:57:48.401979 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.401938    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/013dad5f-f574-48fc-a206-f30aac8de31b-tls-certs\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.401979 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.401983    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-model-cache\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.402217 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.402021    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrj65\" (UniqueName: \"kubernetes.io/projected/013dad5f-f574-48fc-a206-f30aac8de31b-kube-api-access-mrj65\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.402217 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.402049    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-dshm\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.402217 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.402072    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-home\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.402217 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.402132    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-tmp-dir\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.402449 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.402426    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-model-cache\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.402530 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.402515    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-tmp-dir\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.402584 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.402555    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-home\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.404449 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.404430    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-dshm\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.404609 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.404593    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/013dad5f-f574-48fc-a206-f30aac8de31b-tls-certs\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.424814 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.424791    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrj65\" (UniqueName: \"kubernetes.io/projected/013dad5f-f574-48fc-a206-f30aac8de31b-kube-api-access-mrj65\") pod \"rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") " pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.590663 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.590556    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:48.869958 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:48.869927    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"]
Sep 15 21:57:48.873705 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:57:48.873681    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod013dad5f_f574_48fc_a206_f30aac8de31b.slice/crio-bab7fb8878a526b06ee36850d3cb3a5b4f3265a1c08f3c26ea5718fa9e77c9a6 WatchSource:0}: Error finding container bab7fb8878a526b06ee36850d3cb3a5b4f3265a1c08f3c26ea5718fa9e77c9a6: Status 404 returned error can't find the container with id bab7fb8878a526b06ee36850d3cb3a5b4f3265a1c08f3c26ea5718fa9e77c9a6
Sep 15 21:57:49.425188 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:49.425153    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9" event={"ID":"013dad5f-f574-48fc-a206-f30aac8de31b","Type":"ContainerStarted","Data":"65e71d49d9fa1a4ba1127cbcd435672b8bd0b432e0c51cab249f6f7a4d839aeb"}
Sep 15 21:57:49.425606 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:49.425196    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9" event={"ID":"013dad5f-f574-48fc-a206-f30aac8de31b","Type":"ContainerStarted","Data":"bab7fb8878a526b06ee36850d3cb3a5b4f3265a1c08f3c26ea5718fa9e77c9a6"}
Sep 15 21:57:49.455494 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:49.455447    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9" podStartSLOduration=1.455430765 podStartE2EDuration="1.455430765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:57:49.454130475 +0000 UTC m=+1907.266138916" watchObservedRunningTime="2026-09-15 21:57:49.455430765 +0000 UTC m=+1907.267439222"
Sep 15 21:57:53.493087 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.493027    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2" podUID="f21ca1ef-4900-43d9-b3de-07937856363b" containerName="main" containerID="cri-o://76ceb02234c5abae320686319fb847a39bd4ddf6bec3e97f282450c741a200d9" gracePeriod=45
Sep 15 21:57:53.745877 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.745821    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:57:53.851389 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.851357    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f21ca1ef-4900-43d9-b3de-07937856363b-tls-certs\") pod \"f21ca1ef-4900-43d9-b3de-07937856363b\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") "
Sep 15 21:57:53.851581 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.851425    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-tmp-dir\") pod \"f21ca1ef-4900-43d9-b3de-07937856363b\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") "
Sep 15 21:57:53.851581 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.851449    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-model-cache\") pod \"f21ca1ef-4900-43d9-b3de-07937856363b\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") "
Sep 15 21:57:53.851581 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.851481    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7l6z\" (UniqueName: \"kubernetes.io/projected/f21ca1ef-4900-43d9-b3de-07937856363b-kube-api-access-x7l6z\") pod \"f21ca1ef-4900-43d9-b3de-07937856363b\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") "
Sep 15 21:57:53.851581 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.851502    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-home\") pod \"f21ca1ef-4900-43d9-b3de-07937856363b\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") "
Sep 15 21:57:53.851581 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.851546    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-dshm\") pod \"f21ca1ef-4900-43d9-b3de-07937856363b\" (UID: \"f21ca1ef-4900-43d9-b3de-07937856363b\") "
Sep 15 21:57:53.851864 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.851738    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f21ca1ef-4900-43d9-b3de-07937856363b" (UID: "f21ca1ef-4900-43d9-b3de-07937856363b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:57:53.851864 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.851778    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f21ca1ef-4900-43d9-b3de-07937856363b" (UID: "f21ca1ef-4900-43d9-b3de-07937856363b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:57:53.851864 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.851787    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-home" (OuterVolumeSpecName: "home") pod "f21ca1ef-4900-43d9-b3de-07937856363b" (UID: "f21ca1ef-4900-43d9-b3de-07937856363b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:57:53.852010 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.851915    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:57:53.852010 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.851929    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:57:53.852010 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.851937    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:57:53.853747 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.853727    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f21ca1ef-4900-43d9-b3de-07937856363b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f21ca1ef-4900-43d9-b3de-07937856363b" (UID: "f21ca1ef-4900-43d9-b3de-07937856363b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:57:53.853844 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.853725    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-dshm" (OuterVolumeSpecName: "dshm") pod "f21ca1ef-4900-43d9-b3de-07937856363b" (UID: "f21ca1ef-4900-43d9-b3de-07937856363b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:57:53.853844 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.853813    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f21ca1ef-4900-43d9-b3de-07937856363b-kube-api-access-x7l6z" (OuterVolumeSpecName: "kube-api-access-x7l6z") pod "f21ca1ef-4900-43d9-b3de-07937856363b" (UID: "f21ca1ef-4900-43d9-b3de-07937856363b"). InnerVolumeSpecName "kube-api-access-x7l6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:57:53.953399 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.953369    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f21ca1ef-4900-43d9-b3de-07937856363b-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:57:53.953399 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.953393    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f21ca1ef-4900-43d9-b3de-07937856363b-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:57:53.953399 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:53.953404    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7l6z\" (UniqueName: \"kubernetes.io/projected/f21ca1ef-4900-43d9-b3de-07937856363b-kube-api-access-x7l6z\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:57:54.452036 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:54.452001    2575 generic.go:356] "Generic (PLEG): container finished" podID="f21ca1ef-4900-43d9-b3de-07937856363b" containerID="76ceb02234c5abae320686319fb847a39bd4ddf6bec3e97f282450c741a200d9" exitCode=0
Sep 15 21:57:54.452215 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:54.452044    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"
Sep 15 21:57:54.452215 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:54.452070    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2" event={"ID":"f21ca1ef-4900-43d9-b3de-07937856363b","Type":"ContainerDied","Data":"76ceb02234c5abae320686319fb847a39bd4ddf6bec3e97f282450c741a200d9"}
Sep 15 21:57:54.452215 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:54.452109    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2" event={"ID":"f21ca1ef-4900-43d9-b3de-07937856363b","Type":"ContainerDied","Data":"6d05cbcb4cad4323b99fb3df128c7a3d8de9984f33d378b56491c375fb505a5c"}
Sep 15 21:57:54.452215 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:54.452128    2575 scope.go:117] "RemoveContainer" containerID="76ceb02234c5abae320686319fb847a39bd4ddf6bec3e97f282450c741a200d9"
Sep 15 21:57:54.462230 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:54.462214    2575 scope.go:117] "RemoveContainer" containerID="76ceb02234c5abae320686319fb847a39bd4ddf6bec3e97f282450c741a200d9"
Sep 15 21:57:54.462486 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:57:54.462464    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ceb02234c5abae320686319fb847a39bd4ddf6bec3e97f282450c741a200d9\": container with ID starting with 76ceb02234c5abae320686319fb847a39bd4ddf6bec3e97f282450c741a200d9 not found: ID does not exist" containerID="76ceb02234c5abae320686319fb847a39bd4ddf6bec3e97f282450c741a200d9"
Sep 15 21:57:54.462550 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:54.462499    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ceb02234c5abae320686319fb847a39bd4ddf6bec3e97f282450c741a200d9"} err="failed to get container status \"76ceb02234c5abae320686319fb847a39bd4ddf6bec3e97f282450c741a200d9\": rpc error: code = NotFound desc = could not find container \"76ceb02234c5abae320686319fb847a39bd4ddf6bec3e97f282450c741a200d9\": container with ID starting with 76ceb02234c5abae320686319fb847a39bd4ddf6bec3e97f282450c741a200d9 not found: ID does not exist"
Sep 15 21:57:54.477000 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:54.476977    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"]
Sep 15 21:57:54.481892 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:54.481868    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e7a648/sub-test-model-e7a648-kserve-9b7c66c89-fcjl2"]
Sep 15 21:57:54.747216 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:54.747135    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f21ca1ef-4900-43d9-b3de-07937856363b" path="/var/lib/kubelet/pods/f21ca1ef-4900-43d9-b3de-07937856363b/volumes"
Sep 15 21:57:56.829341 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:56.829302    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:57:58.590662 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:58.590630    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:58.591180 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:58.590676    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:58.603423 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:58.603392    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:57:59.488698 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:57:59.488670    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:58:11.941644 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:11.941606    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:58:31.149825 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:31.149786    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:58:46.345294 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:46.345258    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:58:52.229145 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:52.229105    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"]
Sep 15 21:58:53.874253 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:53.874177    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"]
Sep 15 21:58:53.874618 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:53.874571    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f21ca1ef-4900-43d9-b3de-07937856363b" containerName="main"
Sep 15 21:58:53.874618 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:53.874581    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f21ca1ef-4900-43d9-b3de-07937856363b" containerName="main"
Sep 15 21:58:53.874694 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:53.874649    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f21ca1ef-4900-43d9-b3de-07937856363b" containerName="main"
Sep 15 21:58:53.877875 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:53.877857    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:53.882914 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:53.882888    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-0886df-openshift-default-dockercfg-7qbqh\""
Sep 15 21:58:53.883154 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:53.883131    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 21:58:53.903856 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:53.903828    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"]
Sep 15 21:58:54.019497 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.019456    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whwtz\" (UniqueName: \"kubernetes.io/projected/42beaef4-865f-4831-8a1d-30fc885724ff-kube-api-access-whwtz\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.019497 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.019498    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-istio-envoy\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.019748 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.019580    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-istio-data\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.019748 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.019618    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42beaef4-865f-4831-8a1d-30fc885724ff-istio-podinfo\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.019748 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.019647    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-workload-certs\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.019748 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.019715    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42beaef4-865f-4831-8a1d-30fc885724ff-istiod-ca-cert\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.019748 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.019740    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-credential-socket\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.019748 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.019777    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42beaef4-865f-4831-8a1d-30fc885724ff-istio-token\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.019999 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.019802    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-workload-socket\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.120338 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.120296    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42beaef4-865f-4831-8a1d-30fc885724ff-istiod-ca-cert\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.120544 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.120352    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-credential-socket\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.120544 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.120380    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42beaef4-865f-4831-8a1d-30fc885724ff-istio-token\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.120544 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.120414    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-workload-socket\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.120544 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.120469    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whwtz\" (UniqueName: \"kubernetes.io/projected/42beaef4-865f-4831-8a1d-30fc885724ff-kube-api-access-whwtz\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.120544 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.120502    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-istio-envoy\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.120844 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.120562    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-istio-data\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.120844 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.120597    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42beaef4-865f-4831-8a1d-30fc885724ff-istio-podinfo\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.120844 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.120623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-workload-certs\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.120844 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.120748    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-credential-socket\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.121062 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.120950    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-workload-socket\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.121062 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.120977    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-istio-data\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.121062 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.121013    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-workload-certs\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.121162 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.121107    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42beaef4-865f-4831-8a1d-30fc885724ff-istiod-ca-cert\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.123009 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.122974    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-istio-envoy\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.123170 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.123151    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42beaef4-865f-4831-8a1d-30fc885724ff-istio-podinfo\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.133552 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.133492    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42beaef4-865f-4831-8a1d-30fc885724ff-istio-token\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.133635 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.133559    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whwtz\" (UniqueName: \"kubernetes.io/projected/42beaef4-865f-4831-8a1d-30fc885724ff-kube-api-access-whwtz\") pod \"e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") " pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.188778 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.188722    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:54.476968 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.476942    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"]
Sep 15 21:58:54.478903 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:58:54.478875    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42beaef4_865f_4831_8a1d_30fc885724ff.slice/crio-d554d2638585fc0e0c13dfe30254c63735c5854eb8dc9f953e5355814751ed04 WatchSource:0}: Error finding container d554d2638585fc0e0c13dfe30254c63735c5854eb8dc9f953e5355814751ed04: Status 404 returned error can't find the container with id d554d2638585fc0e0c13dfe30254c63735c5854eb8dc9f953e5355814751ed04
Sep 15 21:58:54.480973 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.480943    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:58:54.481038 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.481012    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:58:54.481077 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.481036    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:58:54.773844 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.773740    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp" event={"ID":"42beaef4-865f-4831-8a1d-30fc885724ff","Type":"ContainerStarted","Data":"4b94ea6ae910c0bb89f00b2cfd59b93f05ba21d364091f8ef72351b320ed5480"}
Sep 15 21:58:54.773844 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.773805    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp" event={"ID":"42beaef4-865f-4831-8a1d-30fc885724ff","Type":"ContainerStarted","Data":"d554d2638585fc0e0c13dfe30254c63735c5854eb8dc9f953e5355814751ed04"}
Sep 15 21:58:54.803830 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:54.803780    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp" podStartSLOduration=1.803746493 podStartE2EDuration="1.803746493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:58:54.802320538 +0000 UTC m=+1972.614328994" watchObservedRunningTime="2026-09-15 21:58:54.803746493 +0000 UTC m=+1972.615754937"
Sep 15 21:58:55.189347 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:55.189317    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:55.194229 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:55.194202    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:55.779363 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:55.779329    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:55.780229 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:55.780209    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:58:57.763989 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:58:57.763953    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:59:00.310062 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.310028    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"]
Sep 15 21:59:00.313894 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.313871    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"
Sep 15 21:59:00.322798 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.322772    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0886df\""
Sep 15 21:59:00.339557 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.339533    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"]
Sep 15 21:59:00.435786 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.435734    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"]
Sep 15 21:59:00.439503 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.439486    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"
Sep 15 21:59:00.442543 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.442522    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0886df-dockercfg-9jfxw\""
Sep 15 21:59:00.452326 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.452299    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"]
Sep 15 21:59:00.484741 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.484698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4njhf\" (UniqueName: \"kubernetes.io/projected/ac3b05cb-614a-41c9-8764-53091ff06c83-kube-api-access-4njhf\") pod \"payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879\" (UID: \"ac3b05cb-614a-41c9-8764-53091ff06c83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"
Sep 15 21:59:00.484918 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.484834    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac3b05cb-614a-41c9-8764-53091ff06c83-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879\" (UID: \"ac3b05cb-614a-41c9-8764-53091ff06c83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"
Sep 15 21:59:00.586291 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.586203    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4r9f\" (UniqueName: \"kubernetes.io/projected/44877669-864e-44ff-aefd-9c6344f85d1c-kube-api-access-l4r9f\") pod \"payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng\" (UID: \"44877669-864e-44ff-aefd-9c6344f85d1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"
Sep 15 21:59:00.586291 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.586255    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac3b05cb-614a-41c9-8764-53091ff06c83-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879\" (UID: \"ac3b05cb-614a-41c9-8764-53091ff06c83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"
Sep 15 21:59:00.586486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.586384    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4njhf\" (UniqueName: \"kubernetes.io/projected/ac3b05cb-614a-41c9-8764-53091ff06c83-kube-api-access-4njhf\") pod \"payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879\" (UID: \"ac3b05cb-614a-41c9-8764-53091ff06c83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"
Sep 15 21:59:00.586486 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.586418    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44877669-864e-44ff-aefd-9c6344f85d1c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng\" (UID: \"44877669-864e-44ff-aefd-9c6344f85d1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"
Sep 15 21:59:00.586985 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.586966    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac3b05cb-614a-41c9-8764-53091ff06c83-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879\" (UID: \"ac3b05cb-614a-41c9-8764-53091ff06c83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"
Sep 15 21:59:00.599391 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.599356    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4njhf\" (UniqueName: \"kubernetes.io/projected/ac3b05cb-614a-41c9-8764-53091ff06c83-kube-api-access-4njhf\") pod \"payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879\" (UID: \"ac3b05cb-614a-41c9-8764-53091ff06c83\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"
Sep 15 21:59:00.624498 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.624461    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"
Sep 15 21:59:00.687784 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.687594    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44877669-864e-44ff-aefd-9c6344f85d1c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng\" (UID: \"44877669-864e-44ff-aefd-9c6344f85d1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"
Sep 15 21:59:00.687784 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.687671    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4r9f\" (UniqueName: \"kubernetes.io/projected/44877669-864e-44ff-aefd-9c6344f85d1c-kube-api-access-l4r9f\") pod \"payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng\" (UID: \"44877669-864e-44ff-aefd-9c6344f85d1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"
Sep 15 21:59:00.688455 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.688415    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44877669-864e-44ff-aefd-9c6344f85d1c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng\" (UID: \"44877669-864e-44ff-aefd-9c6344f85d1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"
Sep 15 21:59:00.702282 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.702253    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4r9f\" (UniqueName: \"kubernetes.io/projected/44877669-864e-44ff-aefd-9c6344f85d1c-kube-api-access-l4r9f\") pod \"payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng\" (UID: \"44877669-864e-44ff-aefd-9c6344f85d1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"
Sep 15 21:59:00.751657 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.751628    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"
Sep 15 21:59:00.914166 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:00.914140    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"]
Sep 15 21:59:00.917824 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:59:00.917798    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3b05cb_614a_41c9_8764_53091ff06c83.slice/crio-82ea1e51a4464c5c8a281b04407027d4932d4326fbcaafcfa16486abbfff570b WatchSource:0}: Error finding container 82ea1e51a4464c5c8a281b04407027d4932d4326fbcaafcfa16486abbfff570b: Status 404 returned error can't find the container with id 82ea1e51a4464c5c8a281b04407027d4932d4326fbcaafcfa16486abbfff570b
Sep 15 21:59:01.036364 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:01.036338    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"]
Sep 15 21:59:01.036471 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:59:01.036341    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44877669_864e_44ff_aefd_9c6344f85d1c.slice/crio-c31982cb722029b39fd95150413a1068820ecc4f20a5a7e85db498844aff84d1 WatchSource:0}: Error finding container c31982cb722029b39fd95150413a1068820ecc4f20a5a7e85db498844aff84d1: Status 404 returned error can't find the container with id c31982cb722029b39fd95150413a1068820ecc4f20a5a7e85db498844aff84d1
Sep 15 21:59:01.818934 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:01.818890    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879" event={"ID":"ac3b05cb-614a-41c9-8764-53091ff06c83","Type":"ContainerStarted","Data":"d9be505e40c8dec62e6be25660e39b459e1b1b802900bb441c3b16e7f957ccb4"}
Sep 15 21:59:01.819499 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:01.818956    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879" event={"ID":"ac3b05cb-614a-41c9-8764-53091ff06c83","Type":"ContainerStarted","Data":"82ea1e51a4464c5c8a281b04407027d4932d4326fbcaafcfa16486abbfff570b"}
Sep 15 21:59:01.819499 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:01.818984    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"
Sep 15 21:59:01.820602 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:01.820578    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng" event={"ID":"44877669-864e-44ff-aefd-9c6344f85d1c","Type":"ContainerStarted","Data":"0c630e1f5966a5e5765988a42f8a467d5e455caf89d2ffffc7f169eef1e91d0f"}
Sep 15 21:59:01.820711 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:01.820623    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"
Sep 15 21:59:01.820711 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:01.820638    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng" event={"ID":"44877669-864e-44ff-aefd-9c6344f85d1c","Type":"ContainerStarted","Data":"c31982cb722029b39fd95150413a1068820ecc4f20a5a7e85db498844aff84d1"}
Sep 15 21:59:01.842353 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:01.842303    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879" podStartSLOduration=1.842286908 podStartE2EDuration="1.842286908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:59:01.841524412 +0000 UTC m=+1979.653532857" watchObservedRunningTime="2026-09-15 21:59:01.842286908 +0000 UTC m=+1979.654295389"
Sep 15 21:59:01.869143 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:01.869080    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng" podStartSLOduration=1.8690620340000001 podStartE2EDuration="1.869062034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:59:01.86730514 +0000 UTC m=+1979.679313575" watchObservedRunningTime="2026-09-15 21:59:01.869062034 +0000 UTC m=+1979.681070480"
Sep 15 21:59:03.927564 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:03.927532    2575 scope.go:117] "RemoveContainer" containerID="5063639d297927563a8c89749015c14d672d40b9e6ebe9398c672f23f75b0e9c"
Sep 15 21:59:07.248778 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.248703    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9" podUID="013dad5f-f574-48fc-a206-f30aac8de31b" containerName="main" containerID="cri-o://65e71d49d9fa1a4ba1127cbcd435672b8bd0b432e0c51cab249f6f7a4d839aeb" gracePeriod=45
Sep 15 21:59:07.529041 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.529015    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:59:07.578219 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.578183    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-home\") pod \"013dad5f-f574-48fc-a206-f30aac8de31b\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") "
Sep 15 21:59:07.578417 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.578240    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-tmp-dir\") pod \"013dad5f-f574-48fc-a206-f30aac8de31b\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") "
Sep 15 21:59:07.578417 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.578279    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrj65\" (UniqueName: \"kubernetes.io/projected/013dad5f-f574-48fc-a206-f30aac8de31b-kube-api-access-mrj65\") pod \"013dad5f-f574-48fc-a206-f30aac8de31b\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") "
Sep 15 21:59:07.578417 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.578343    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-model-cache\") pod \"013dad5f-f574-48fc-a206-f30aac8de31b\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") "
Sep 15 21:59:07.578417 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.578370    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/013dad5f-f574-48fc-a206-f30aac8de31b-tls-certs\") pod \"013dad5f-f574-48fc-a206-f30aac8de31b\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") "
Sep 15 21:59:07.578646 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.578430    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-dshm\") pod \"013dad5f-f574-48fc-a206-f30aac8de31b\" (UID: \"013dad5f-f574-48fc-a206-f30aac8de31b\") "
Sep 15 21:59:07.579110 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.578974    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-home" (OuterVolumeSpecName: "home") pod "013dad5f-f574-48fc-a206-f30aac8de31b" (UID: "013dad5f-f574-48fc-a206-f30aac8de31b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:59:07.579229 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.579133    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "013dad5f-f574-48fc-a206-f30aac8de31b" (UID: "013dad5f-f574-48fc-a206-f30aac8de31b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:59:07.579296 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.579269    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-model-cache" (OuterVolumeSpecName: "model-cache") pod "013dad5f-f574-48fc-a206-f30aac8de31b" (UID: "013dad5f-f574-48fc-a206-f30aac8de31b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:59:07.581901 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.581870    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-dshm" (OuterVolumeSpecName: "dshm") pod "013dad5f-f574-48fc-a206-f30aac8de31b" (UID: "013dad5f-f574-48fc-a206-f30aac8de31b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:59:07.582056 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.582031    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/013dad5f-f574-48fc-a206-f30aac8de31b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "013dad5f-f574-48fc-a206-f30aac8de31b" (UID: "013dad5f-f574-48fc-a206-f30aac8de31b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:59:07.582168 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.582127    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/013dad5f-f574-48fc-a206-f30aac8de31b-kube-api-access-mrj65" (OuterVolumeSpecName: "kube-api-access-mrj65") pod "013dad5f-f574-48fc-a206-f30aac8de31b" (UID: "013dad5f-f574-48fc-a206-f30aac8de31b"). InnerVolumeSpecName "kube-api-access-mrj65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:59:07.679313 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.679276    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrj65\" (UniqueName: \"kubernetes.io/projected/013dad5f-f574-48fc-a206-f30aac8de31b-kube-api-access-mrj65\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:59:07.679313 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.679305    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:59:07.679313 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.679315    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/013dad5f-f574-48fc-a206-f30aac8de31b-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:59:07.679313 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.679322    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:59:07.679575 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.679332    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:59:07.679575 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.679339    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/013dad5f-f574-48fc-a206-f30aac8de31b-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 21:59:07.859043 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.855468    2575 generic.go:356] "Generic (PLEG): container finished" podID="013dad5f-f574-48fc-a206-f30aac8de31b" containerID="65e71d49d9fa1a4ba1127cbcd435672b8bd0b432e0c51cab249f6f7a4d839aeb" exitCode=0
Sep 15 21:59:07.859043 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.855621    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9" event={"ID":"013dad5f-f574-48fc-a206-f30aac8de31b","Type":"ContainerDied","Data":"65e71d49d9fa1a4ba1127cbcd435672b8bd0b432e0c51cab249f6f7a4d839aeb"}
Sep 15 21:59:07.859043 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.855659    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9" event={"ID":"013dad5f-f574-48fc-a206-f30aac8de31b","Type":"ContainerDied","Data":"bab7fb8878a526b06ee36850d3cb3a5b4f3265a1c08f3c26ea5718fa9e77c9a6"}
Sep 15 21:59:07.859043 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.855681    2575 scope.go:117] "RemoveContainer" containerID="65e71d49d9fa1a4ba1127cbcd435672b8bd0b432e0c51cab249f6f7a4d839aeb"
Sep 15 21:59:07.859043 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.855886    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"
Sep 15 21:59:07.873206 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.873090    2575 scope.go:117] "RemoveContainer" containerID="65e71d49d9fa1a4ba1127cbcd435672b8bd0b432e0c51cab249f6f7a4d839aeb"
Sep 15 21:59:07.873770 ip-10-0-135-148 kubenswrapper[2575]: E0915 21:59:07.873658    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e71d49d9fa1a4ba1127cbcd435672b8bd0b432e0c51cab249f6f7a4d839aeb\": container with ID starting with 65e71d49d9fa1a4ba1127cbcd435672b8bd0b432e0c51cab249f6f7a4d839aeb not found: ID does not exist" containerID="65e71d49d9fa1a4ba1127cbcd435672b8bd0b432e0c51cab249f6f7a4d839aeb"
Sep 15 21:59:07.873770 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.873712    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e71d49d9fa1a4ba1127cbcd435672b8bd0b432e0c51cab249f6f7a4d839aeb"} err="failed to get container status \"65e71d49d9fa1a4ba1127cbcd435672b8bd0b432e0c51cab249f6f7a4d839aeb\": rpc error: code = NotFound desc = could not find container \"65e71d49d9fa1a4ba1127cbcd435672b8bd0b432e0c51cab249f6f7a4d839aeb\": container with ID starting with 65e71d49d9fa1a4ba1127cbcd435672b8bd0b432e0c51cab249f6f7a4d839aeb not found: ID does not exist"
Sep 15 21:59:07.897722 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.897682    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"]
Sep 15 21:59:07.903739 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:07.903714    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ce3ab0/rate-test-model-ce3ab0-kserve-6d97656b7-4mhx9"]
Sep 15 21:59:08.747341 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:08.747300    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="013dad5f-f574-48fc-a206-f30aac8de31b" path="/var/lib/kubelet/pods/013dad5f-f574-48fc-a206-f30aac8de31b/volumes"
Sep 15 21:59:10.754401 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:10.754368    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:59:12.828453 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:12.828425    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"
Sep 15 21:59:12.828943 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:12.828502    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"
Sep 15 21:59:14.910570 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:14.910536    2575 generic.go:356] "Generic (PLEG): container finished" podID="42beaef4-865f-4831-8a1d-30fc885724ff" containerID="4b94ea6ae910c0bb89f00b2cfd59b93f05ba21d364091f8ef72351b320ed5480" exitCode=0
Sep 15 21:59:14.911012 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:14.910602    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp" event={"ID":"42beaef4-865f-4831-8a1d-30fc885724ff","Type":"ContainerDied","Data":"4b94ea6ae910c0bb89f00b2cfd59b93f05ba21d364091f8ef72351b320ed5480"}
Sep 15 21:59:14.911074 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:14.911041    2575 scope.go:117] "RemoveContainer" containerID="4b94ea6ae910c0bb89f00b2cfd59b93f05ba21d364091f8ef72351b320ed5480"
Sep 15 21:59:14.912038 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:14.912013    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:59:14.912118 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:14.912076    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:59:14.912118 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:14.912105    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:59:15.918247 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:15.918207    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp" event={"ID":"42beaef4-865f-4831-8a1d-30fc885724ff","Type":"ContainerStarted","Data":"3885d24171d6c3acaf822823e390984e1df841741158730f2fa392b67adbf7cd"}
Sep 15 21:59:16.189876 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:16.189771    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:59:16.190034 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:16.189998    2575 patch_prober.go:28] interesting pod/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Sep 15 21:59:16.190078 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:16.190044    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp" podUID="42beaef4-865f-4831-8a1d-30fc885724ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Sep 15 21:59:17.190090 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:17.190051    2575 patch_prober.go:28] interesting pod/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Sep 15 21:59:17.190491 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:17.190125    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp" podUID="42beaef4-865f-4831-8a1d-30fc885724ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Sep 15 21:59:18.189208 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:18.189176    2575 patch_prober.go:28] interesting pod/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Sep 15 21:59:18.189380 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:18.189226    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp" podUID="42beaef4-865f-4831-8a1d-30fc885724ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Sep 15 21:59:19.189562 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:19.189523    2575 patch_prober.go:28] interesting pod/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Sep 15 21:59:19.190012 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:19.189585    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp" podUID="42beaef4-865f-4831-8a1d-30fc885724ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Sep 15 21:59:20.189517 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:20.189481    2575 patch_prober.go:28] interesting pod/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Sep 15 21:59:20.189690 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:20.189538    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp" podUID="42beaef4-865f-4831-8a1d-30fc885724ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Sep 15 21:59:21.193652 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:21.193618    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:59:21.194068 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:21.194007    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:59:21.194783 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:21.194743    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 21:59:28.053169 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.053136    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"]
Sep 15 21:59:28.053565 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.053553    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="013dad5f-f574-48fc-a206-f30aac8de31b" containerName="main"
Sep 15 21:59:28.053565 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.053566    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="013dad5f-f574-48fc-a206-f30aac8de31b" containerName="main"
Sep 15 21:59:28.053649 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.053629    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="013dad5f-f574-48fc-a206-f30aac8de31b" containerName="main"
Sep 15 21:59:28.057021 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.057006    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.067595 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.067577    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1297a3-openshift-default-dockercfg-sp47h\""
Sep 15 21:59:28.081612 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.081586    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"]
Sep 15 21:59:28.179260 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.179220    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istiod-ca-cert\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.179260 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.179266    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-data\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.179488 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.179282    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-podinfo\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.179488 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.179370    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-credential-socket\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.179488 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.179412    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-workload-certs\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.179488 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.179447    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-envoy\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.179488 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.179471    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j25c\" (UniqueName: \"kubernetes.io/projected/bb690595-954f-4b8d-b5ee-ca7af3779ac2-kube-api-access-8j25c\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.179666 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.179506    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-workload-socket\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.179666 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.179556    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-token\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.281056 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.281017    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-token\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.281247 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.281077    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istiod-ca-cert\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.281247 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.281107    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-data\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.281247 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.281126    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-podinfo\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.281247 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.281150    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-credential-socket\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.281505 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.281256    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-workload-certs\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.281505 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.281304    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-envoy\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.281505 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.281334    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j25c\" (UniqueName: \"kubernetes.io/projected/bb690595-954f-4b8d-b5ee-ca7af3779ac2-kube-api-access-8j25c\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.281505 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.281381    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-workload-socket\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.281726 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.281569    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-data\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.281831 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.281800    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-credential-socket\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.281893 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.281852    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-workload-certs\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.281943 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.281896    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-workload-socket\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.282089 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.282071    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istiod-ca-cert\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.283635 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.283614    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-envoy\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.283943 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.283925    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-podinfo\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.294188 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.294165    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-token\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.295067 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.295048    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j25c\" (UniqueName: \"kubernetes.io/projected/bb690595-954f-4b8d-b5ee-ca7af3779ac2-kube-api-access-8j25c\") pod \"e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") " pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.367272 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.367242    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:28.653553 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.653524    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"]
Sep 15 21:59:28.654405 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:59:28.654378    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb690595_954f_4b8d_b5ee_ca7af3779ac2.slice/crio-4119866ac7fa8c2c2727b5c386bac8ac64a223252480629478202f419c04053b WatchSource:0}: Error finding container 4119866ac7fa8c2c2727b5c386bac8ac64a223252480629478202f419c04053b: Status 404 returned error can't find the container with id 4119866ac7fa8c2c2727b5c386bac8ac64a223252480629478202f419c04053b
Sep 15 21:59:28.656999 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.656968    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:59:28.657083 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.657036    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:59:28.657083 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.657065    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:59:28.989198 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.989110    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg" event={"ID":"bb690595-954f-4b8d-b5ee-ca7af3779ac2","Type":"ContainerStarted","Data":"b2f92cd2ae46616efcb701cd226359a02e0bb50874145b39a596e50f217f37f9"}
Sep 15 21:59:28.989198 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:28.989168    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg" event={"ID":"bb690595-954f-4b8d-b5ee-ca7af3779ac2","Type":"ContainerStarted","Data":"4119866ac7fa8c2c2727b5c386bac8ac64a223252480629478202f419c04053b"}
Sep 15 21:59:29.367685 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:29.367651    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:29.372527 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:29.372507    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:29.405964 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:29.405920    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg" podStartSLOduration=1.405906112 podStartE2EDuration="1.405906112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:59:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:59:29.052748416 +0000 UTC m=+2006.864756859" watchObservedRunningTime="2026-09-15 21:59:29.405906112 +0000 UTC m=+2007.217914553"
Sep 15 21:59:29.995098 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:29.995058    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:29.996101 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:29.996078    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:35.639871 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.639838    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"]
Sep 15 21:59:35.643410 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.643394    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"
Sep 15 21:59:35.645896 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.645875    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1297a3\""
Sep 15 21:59:35.646127 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.646112    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1297a3-dockercfg-87fzh\""
Sep 15 21:59:35.665333 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.665313    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"]
Sep 15 21:59:35.754089 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.754043    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f095e5-a6e0-4fff-9ccb-36ded38233a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl\" (UID: \"17f095e5-a6e0-4fff-9ccb-36ded38233a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"
Sep 15 21:59:35.754281 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.754101    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpdll\" (UniqueName: \"kubernetes.io/projected/17f095e5-a6e0-4fff-9ccb-36ded38233a1-kube-api-access-rpdll\") pod \"payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl\" (UID: \"17f095e5-a6e0-4fff-9ccb-36ded38233a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"
Sep 15 21:59:35.766405 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.766369    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"]
Sep 15 21:59:35.770013 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.769991    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"
Sep 15 21:59:35.785554 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.785523    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"]
Sep 15 21:59:35.854763 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.854714    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f095e5-a6e0-4fff-9ccb-36ded38233a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl\" (UID: \"17f095e5-a6e0-4fff-9ccb-36ded38233a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"
Sep 15 21:59:35.854932 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.854777    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpdll\" (UniqueName: \"kubernetes.io/projected/17f095e5-a6e0-4fff-9ccb-36ded38233a1-kube-api-access-rpdll\") pod \"payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl\" (UID: \"17f095e5-a6e0-4fff-9ccb-36ded38233a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"
Sep 15 21:59:35.854932 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.854800    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjd4v\" (UniqueName: \"kubernetes.io/projected/79c17a0d-49c8-4126-88a2-664e666d84d0-kube-api-access-xjd4v\") pod \"payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz\" (UID: \"79c17a0d-49c8-4126-88a2-664e666d84d0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"
Sep 15 21:59:35.854932 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.854874    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c17a0d-49c8-4126-88a2-664e666d84d0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz\" (UID: \"79c17a0d-49c8-4126-88a2-664e666d84d0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"
Sep 15 21:59:35.855430 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.855410    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f095e5-a6e0-4fff-9ccb-36ded38233a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl\" (UID: \"17f095e5-a6e0-4fff-9ccb-36ded38233a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"
Sep 15 21:59:35.863233 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.863206    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpdll\" (UniqueName: \"kubernetes.io/projected/17f095e5-a6e0-4fff-9ccb-36ded38233a1-kube-api-access-rpdll\") pod \"payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl\" (UID: \"17f095e5-a6e0-4fff-9ccb-36ded38233a1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"
Sep 15 21:59:35.954227 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.954144    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"
Sep 15 21:59:35.955478 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.955454    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c17a0d-49c8-4126-88a2-664e666d84d0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz\" (UID: \"79c17a0d-49c8-4126-88a2-664e666d84d0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"
Sep 15 21:59:35.955542 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.955529    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjd4v\" (UniqueName: \"kubernetes.io/projected/79c17a0d-49c8-4126-88a2-664e666d84d0-kube-api-access-xjd4v\") pod \"payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz\" (UID: \"79c17a0d-49c8-4126-88a2-664e666d84d0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"
Sep 15 21:59:35.956156 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.956138    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c17a0d-49c8-4126-88a2-664e666d84d0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz\" (UID: \"79c17a0d-49c8-4126-88a2-664e666d84d0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"
Sep 15 21:59:35.966609 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:35.966585    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjd4v\" (UniqueName: \"kubernetes.io/projected/79c17a0d-49c8-4126-88a2-664e666d84d0-kube-api-access-xjd4v\") pod \"payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz\" (UID: \"79c17a0d-49c8-4126-88a2-664e666d84d0\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"
Sep 15 21:59:36.080485 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:36.080451    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"
Sep 15 21:59:36.231451 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:36.231426    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"]
Sep 15 21:59:36.233071 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:59:36.233047    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f095e5_a6e0_4fff_9ccb_36ded38233a1.slice/crio-556d1f946757b940fe07fdc5e7a128b9ba384f4567e8630d145c3070cc85cbf7 WatchSource:0}: Error finding container 556d1f946757b940fe07fdc5e7a128b9ba384f4567e8630d145c3070cc85cbf7: Status 404 returned error can't find the container with id 556d1f946757b940fe07fdc5e7a128b9ba384f4567e8630d145c3070cc85cbf7
Sep 15 21:59:36.355122 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:36.355094    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"]
Sep 15 21:59:36.356709 ip-10-0-135-148 kubenswrapper[2575]: W0915 21:59:36.356673    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c17a0d_49c8_4126_88a2_664e666d84d0.slice/crio-2e6254f17060ea64cc7f449bdbb929c8027d0ededd835d39b503268003069269 WatchSource:0}: Error finding container 2e6254f17060ea64cc7f449bdbb929c8027d0ededd835d39b503268003069269: Status 404 returned error can't find the container with id 2e6254f17060ea64cc7f449bdbb929c8027d0ededd835d39b503268003069269
Sep 15 21:59:37.035294 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:37.035251    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz" event={"ID":"79c17a0d-49c8-4126-88a2-664e666d84d0","Type":"ContainerStarted","Data":"e52c4c395ffda85565bb548625a0481b5d39a667054facc27a87b17ca57da657"}
Sep 15 21:59:37.035784 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:37.035347    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz" event={"ID":"79c17a0d-49c8-4126-88a2-664e666d84d0","Type":"ContainerStarted","Data":"2e6254f17060ea64cc7f449bdbb929c8027d0ededd835d39b503268003069269"}
Sep 15 21:59:37.035784 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:37.035383    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"
Sep 15 21:59:37.036881 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:37.036857    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl" event={"ID":"17f095e5-a6e0-4fff-9ccb-36ded38233a1","Type":"ContainerStarted","Data":"8d3f556a107446c3ad774aa78b46fec01c47beeef4d2daa04a0420196a25a785"}
Sep 15 21:59:37.036959 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:37.036903    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl" event={"ID":"17f095e5-a6e0-4fff-9ccb-36ded38233a1","Type":"ContainerStarted","Data":"556d1f946757b940fe07fdc5e7a128b9ba384f4567e8630d145c3070cc85cbf7"}
Sep 15 21:59:37.036959 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:37.036944    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"
Sep 15 21:59:37.061920 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:37.061870    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz" podStartSLOduration=2.061854052 podStartE2EDuration="2.061854052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:59:37.060232256 +0000 UTC m=+2014.872240705" watchObservedRunningTime="2026-09-15 21:59:37.061854052 +0000 UTC m=+2014.873862496"
Sep 15 21:59:37.079157 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:37.079096    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl" podStartSLOduration=2.079077084 podStartE2EDuration="2.079077084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:59:37.076306384 +0000 UTC m=+2014.888314829" watchObservedRunningTime="2026-09-15 21:59:37.079077084 +0000 UTC m=+2014.891085530"
Sep 15 21:59:39.251261 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:39.251227    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 21:59:48.044861 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:48.044826    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"
Sep 15 21:59:48.045344 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:48.045133    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"
Sep 15 21:59:50.117073 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:50.117038    2575 generic.go:356] "Generic (PLEG): container finished" podID="bb690595-954f-4b8d-b5ee-ca7af3779ac2" containerID="b2f92cd2ae46616efcb701cd226359a02e0bb50874145b39a596e50f217f37f9" exitCode=0
Sep 15 21:59:50.117598 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:50.117111    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg" event={"ID":"bb690595-954f-4b8d-b5ee-ca7af3779ac2","Type":"ContainerDied","Data":"b2f92cd2ae46616efcb701cd226359a02e0bb50874145b39a596e50f217f37f9"}
Sep 15 21:59:50.117696 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:50.117618    2575 scope.go:117] "RemoveContainer" containerID="b2f92cd2ae46616efcb701cd226359a02e0bb50874145b39a596e50f217f37f9"
Sep 15 21:59:50.118788 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:50.118744    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:59:50.118864 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:50.118829    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:59:50.118902 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:50.118867    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 21:59:51.124493 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:51.124458    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg" event={"ID":"bb690595-954f-4b8d-b5ee-ca7af3779ac2","Type":"ContainerStarted","Data":"031c1f90aa8155799e1c0fc7214f769acd45b43d836e7fcd3d965a0862be7ce2"}
Sep 15 21:59:51.368112 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:51.368065    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:51.368528 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:51.368495    2575 patch_prober.go:28] interesting pod/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Sep 15 21:59:51.368661 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:51.368557    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg" podUID="bb690595-954f-4b8d-b5ee-ca7af3779ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Sep 15 21:59:52.368587 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:52.368107    2575 patch_prober.go:28] interesting pod/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Sep 15 21:59:52.368587 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:52.368176    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg" podUID="bb690595-954f-4b8d-b5ee-ca7af3779ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Sep 15 21:59:53.367786 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:53.367729    2575 patch_prober.go:28] interesting pod/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Sep 15 21:59:53.368085 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:53.368053    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg" podUID="bb690595-954f-4b8d-b5ee-ca7af3779ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Sep 15 21:59:54.367651 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:54.367613    2575 patch_prober.go:28] interesting pod/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Sep 15 21:59:54.368278 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:54.368246    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg" podUID="bb690595-954f-4b8d-b5ee-ca7af3779ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Sep 15 21:59:55.368297 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:55.368254    2575 patch_prober.go:28] interesting pod/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Sep 15 21:59:55.368985 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:55.368952    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg" podUID="bb690595-954f-4b8d-b5ee-ca7af3779ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Sep 15 21:59:56.373944 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:56.373910    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:56.374875 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:56.374703    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 21:59:56.375841 ip-10-0-135-148 kubenswrapper[2575]: I0915 21:59:56.375821    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 22:00:00.156371 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:00.156336    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce3ab0-29825160-kjx92"]
Sep 15 22:00:00.160152 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:00.160130    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce3ab0-29825160-kjx92"
Sep 15 22:00:00.163504 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:00.163467    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 22:00:00.163982 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:00.163958    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e7a648-29825160-cv4p8"]
Sep 15 22:00:00.169417 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:00.169397    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e7a648-29825160-cv4p8"
Sep 15 22:00:00.173882 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:00.173859    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce3ab0-29825160-kjx92"
Sep 15 22:00:00.175211 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:00.175191    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1297a3-29825160-6g8jd"]
Sep 15 22:00:00.178970 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:00.178951    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1297a3-29825160-6g8jd"
Sep 15 22:00:00.183486 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:00.183338    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce3ab0-29825160-kjx92"]
Sep 15 22:00:00.187384 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:00.187361    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e7a648-29825160-cv4p8"]
Sep 15 22:00:00.209149 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:00.209115    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e7a648-29825160-cv4p8"
Sep 15 22:00:00.215353 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:00.215327    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1297a3-29825160-6g8jd"
Sep 15 22:00:00.223826 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:00.223112    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1297a3-29825160-6g8jd"]
Sep 15 22:00:00.470708 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:00.470684    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce3ab0-29825160-kjx92"]
Sep 15 22:00:00.472969 ip-10-0-135-148 kubenswrapper[2575]: W0915 22:00:00.472943    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84691e5_184c_4031_a84c_481afcc111b3.slice/crio-c7a35e92b65f6c8d691eccee8240d07f274f28627b99c9e1650ed1af2814eae2 WatchSource:0}: Error finding container c7a35e92b65f6c8d691eccee8240d07f274f28627b99c9e1650ed1af2814eae2: Status 404 returned error can't find the container with id c7a35e92b65f6c8d691eccee8240d07f274f28627b99c9e1650ed1af2814eae2
Sep 15 22:00:00.509206 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:00.509163    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e7a648-29825160-cv4p8"]
Sep 15 22:00:00.511850 ip-10-0-135-148 kubenswrapper[2575]: W0915 22:00:00.511819    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9050b59e_6373_4ae9_8c2a_5cfe8d7be911.slice/crio-1c3e51f13219b2ffe73130f24f2de9e123a258d1e4056af261c660934269cdcb WatchSource:0}: Error finding container 1c3e51f13219b2ffe73130f24f2de9e123a258d1e4056af261c660934269cdcb: Status 404 returned error can't find the container with id 1c3e51f13219b2ffe73130f24f2de9e123a258d1e4056af261c660934269cdcb
Sep 15 22:00:00.526450 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:00.526420    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1297a3-29825160-6g8jd"]
Sep 15 22:00:00.530743 ip-10-0-135-148 kubenswrapper[2575]: W0915 22:00:00.530720    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3a1b68_1489_44eb_b26d_45cb64f4711e.slice/crio-a321738919321a1a10603a9c2d627e5317de6330f144d051ebd7f68a1da344ee WatchSource:0}: Error finding container a321738919321a1a10603a9c2d627e5317de6330f144d051ebd7f68a1da344ee: Status 404 returned error can't find the container with id a321738919321a1a10603a9c2d627e5317de6330f144d051ebd7f68a1da344ee
Sep 15 22:00:01.185248 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:01.185211    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce3ab0-29825160-kjx92" event={"ID":"f84691e5-184c-4031-a84c-481afcc111b3","Type":"ContainerStarted","Data":"0198efc69ee803da6b21dd9c9f1088a397c5c854f3fd34df866abbe46fc052c2"}
Sep 15 22:00:01.185723 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:01.185274    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce3ab0-29825160-kjx92" event={"ID":"f84691e5-184c-4031-a84c-481afcc111b3","Type":"ContainerStarted","Data":"c7a35e92b65f6c8d691eccee8240d07f274f28627b99c9e1650ed1af2814eae2"}
Sep 15 22:00:01.186960 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:01.186936    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1297a3-29825160-6g8jd" event={"ID":"ec3a1b68-1489-44eb-b26d-45cb64f4711e","Type":"ContainerStarted","Data":"e25ad19a56e4d6df5164017fe901231fbf1713453b60df74a22aa1f6cac5e8e7"}
Sep 15 22:00:01.187064 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:01.186981    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1297a3-29825160-6g8jd" event={"ID":"ec3a1b68-1489-44eb-b26d-45cb64f4711e","Type":"ContainerStarted","Data":"a321738919321a1a10603a9c2d627e5317de6330f144d051ebd7f68a1da344ee"}
Sep 15 22:00:01.188451 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:01.188433    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e7a648-29825160-cv4p8" event={"ID":"9050b59e-6373-4ae9-8c2a-5cfe8d7be911","Type":"ContainerStarted","Data":"315a417a00c816f30f2c58aef2cd1060467942e1a9de491fbbb63ce4fccff474"}
Sep 15 22:00:01.188539 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:01.188466    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e7a648-29825160-cv4p8" event={"ID":"9050b59e-6373-4ae9-8c2a-5cfe8d7be911","Type":"ContainerStarted","Data":"1c3e51f13219b2ffe73130f24f2de9e123a258d1e4056af261c660934269cdcb"}
Sep 15 22:00:01.215952 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:01.215885    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce3ab0-29825160-kjx92" podStartSLOduration=1.215863571 podStartE2EDuration="1.215863571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:00:01.214982299 +0000 UTC m=+2039.026990742" watchObservedRunningTime="2026-09-15 22:00:01.215863571 +0000 UTC m=+2039.027872017"
Sep 15 22:00:01.243916 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:01.243862    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1297a3-29825160-6g8jd" podStartSLOduration=1.243846189 podStartE2EDuration="1.243846189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:00:01.2423783 +0000 UTC m=+2039.054386745" watchObservedRunningTime="2026-09-15 22:00:01.243846189 +0000 UTC m=+2039.055854632"
Sep 15 22:00:01.269835 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:01.269783    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e7a648-29825160-cv4p8" podStartSLOduration=1.26974073 podStartE2EDuration="1.26974073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:00:01.268947333 +0000 UTC m=+2039.080955777" watchObservedRunningTime="2026-09-15 22:00:01.26974073 +0000 UTC m=+2039.081749218"
Sep 15 22:00:02.195043 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:02.195005    2575 generic.go:356] "Generic (PLEG): container finished" podID="ec3a1b68-1489-44eb-b26d-45cb64f4711e" containerID="e25ad19a56e4d6df5164017fe901231fbf1713453b60df74a22aa1f6cac5e8e7" exitCode=0
Sep 15 22:00:02.195474 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:02.195101    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1297a3-29825160-6g8jd" event={"ID":"ec3a1b68-1489-44eb-b26d-45cb64f4711e","Type":"ContainerDied","Data":"e25ad19a56e4d6df5164017fe901231fbf1713453b60df74a22aa1f6cac5e8e7"}
Sep 15 22:00:02.196539 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:02.196516    2575 generic.go:356] "Generic (PLEG): container finished" podID="9050b59e-6373-4ae9-8c2a-5cfe8d7be911" containerID="315a417a00c816f30f2c58aef2cd1060467942e1a9de491fbbb63ce4fccff474" exitCode=0
Sep 15 22:00:02.196656 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:02.196596    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e7a648-29825160-cv4p8" event={"ID":"9050b59e-6373-4ae9-8c2a-5cfe8d7be911","Type":"ContainerDied","Data":"315a417a00c816f30f2c58aef2cd1060467942e1a9de491fbbb63ce4fccff474"}
Sep 15 22:00:02.197962 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:02.197937    2575 generic.go:356] "Generic (PLEG): container finished" podID="f84691e5-184c-4031-a84c-481afcc111b3" containerID="0198efc69ee803da6b21dd9c9f1088a397c5c854f3fd34df866abbe46fc052c2" exitCode=0
Sep 15 22:00:02.198070 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:02.197976    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce3ab0-29825160-kjx92" event={"ID":"f84691e5-184c-4031-a84c-481afcc111b3","Type":"ContainerDied","Data":"0198efc69ee803da6b21dd9c9f1088a397c5c854f3fd34df866abbe46fc052c2"}
Sep 15 22:00:03.346720 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:03.346688    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce3ab0-29825160-kjx92"
Sep 15 22:00:03.396815 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:03.396790    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1297a3-29825160-6g8jd"
Sep 15 22:00:03.400088 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:03.400059    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e7a648-29825160-cv4p8"
Sep 15 22:00:03.948394 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:03.948359    2575 scope.go:117] "RemoveContainer" containerID="c163f9d031c94b6fddf452aa30529674d3527f7a9efd0ccc8caceb48920ac8ea"
Sep 15 22:00:03.957276 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:03.957257    2575 scope.go:117] "RemoveContainer" containerID="5b1ab5720f3489a8250ba5646e3ace08592005817e7fb47a48fb556161867013"
Sep 15 22:00:04.210566 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:04.210479    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e7a648-29825160-cv4p8"
Sep 15 22:00:04.210566 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:04.210523    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e7a648-29825160-cv4p8" event={"ID":"9050b59e-6373-4ae9-8c2a-5cfe8d7be911","Type":"ContainerDied","Data":"1c3e51f13219b2ffe73130f24f2de9e123a258d1e4056af261c660934269cdcb"}
Sep 15 22:00:04.210773 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:04.210586    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c3e51f13219b2ffe73130f24f2de9e123a258d1e4056af261c660934269cdcb"
Sep 15 22:00:04.211858 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:04.211836    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce3ab0-29825160-kjx92"
Sep 15 22:00:04.211974 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:04.211876    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce3ab0-29825160-kjx92" event={"ID":"f84691e5-184c-4031-a84c-481afcc111b3","Type":"ContainerDied","Data":"c7a35e92b65f6c8d691eccee8240d07f274f28627b99c9e1650ed1af2814eae2"}
Sep 15 22:00:04.211974 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:04.211918    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7a35e92b65f6c8d691eccee8240d07f274f28627b99c9e1650ed1af2814eae2"
Sep 15 22:00:04.213048 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:04.213029    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1297a3-29825160-6g8jd" event={"ID":"ec3a1b68-1489-44eb-b26d-45cb64f4711e","Type":"ContainerDied","Data":"a321738919321a1a10603a9c2d627e5317de6330f144d051ebd7f68a1da344ee"}
Sep 15 22:00:04.213150 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:04.213061    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a321738919321a1a10603a9c2d627e5317de6330f144d051ebd7f68a1da344ee"
Sep 15 22:00:04.213150 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:04.213067    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1297a3-29825160-6g8jd"
Sep 15 22:00:06.413816 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:06.413787    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz_79c17a0d-49c8-4126-88a2-664e666d84d0/payload-processing/0.log"
Sep 15 22:00:15.245644 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:15.245606    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:00:30.952783 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:30.952686    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:00:37.341352 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:37.341321    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng_44877669-864e-44ff-aefd-9c6344f85d1c/payload-processing/0.log"
Sep 15 22:00:37.577770 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:37.577733    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz_79c17a0d-49c8-4126-88a2-664e666d84d0/payload-processing/0.log"
Sep 15 22:00:37.697459 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:37.697435    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz_79c17a0d-49c8-4126-88a2-664e666d84d0/payload-processing/0.log"
Sep 15 22:00:47.285576 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.285538    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"]
Sep 15 22:00:47.286029 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.286014    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec3a1b68-1489-44eb-b26d-45cb64f4711e" containerName="cleanup"
Sep 15 22:00:47.286029 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.286030    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec3a1b68-1489-44eb-b26d-45cb64f4711e" containerName="cleanup"
Sep 15 22:00:47.286106 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.286051    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f84691e5-184c-4031-a84c-481afcc111b3" containerName="cleanup"
Sep 15 22:00:47.286106 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.286057    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f84691e5-184c-4031-a84c-481afcc111b3" containerName="cleanup"
Sep 15 22:00:47.286106 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.286070    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9050b59e-6373-4ae9-8c2a-5cfe8d7be911" containerName="cleanup"
Sep 15 22:00:47.286106 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.286075    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9050b59e-6373-4ae9-8c2a-5cfe8d7be911" containerName="cleanup"
Sep 15 22:00:47.286231 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.286128    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec3a1b68-1489-44eb-b26d-45cb64f4711e" containerName="cleanup"
Sep 15 22:00:47.286231 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.286140    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f84691e5-184c-4031-a84c-481afcc111b3" containerName="cleanup"
Sep 15 22:00:47.286231 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.286148    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9050b59e-6373-4ae9-8c2a-5cfe8d7be911" containerName="cleanup"
Sep 15 22:00:47.289398 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.289379    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"
Sep 15 22:00:47.291947 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.291924    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-3596d6-dockercfg-xswsx\""
Sep 15 22:00:47.292067 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.291997    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-3596d6\""
Sep 15 22:00:47.305594 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.305561    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"]
Sep 15 22:00:47.357332 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.357300    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3eabd21-de32-4922-b790-135ec968ff65-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p\" (UID: \"e3eabd21-de32-4922-b790-135ec968ff65\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"
Sep 15 22:00:47.357524 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.357343    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7kt7\" (UniqueName: \"kubernetes.io/projected/e3eabd21-de32-4922-b790-135ec968ff65-kube-api-access-x7kt7\") pod \"payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p\" (UID: \"e3eabd21-de32-4922-b790-135ec968ff65\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"
Sep 15 22:00:47.458688 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.458652    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3eabd21-de32-4922-b790-135ec968ff65-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p\" (UID: \"e3eabd21-de32-4922-b790-135ec968ff65\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"
Sep 15 22:00:47.458688 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.458693    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7kt7\" (UniqueName: \"kubernetes.io/projected/e3eabd21-de32-4922-b790-135ec968ff65-kube-api-access-x7kt7\") pod \"payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p\" (UID: \"e3eabd21-de32-4922-b790-135ec968ff65\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"
Sep 15 22:00:47.459460 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.459442    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3eabd21-de32-4922-b790-135ec968ff65-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p\" (UID: \"e3eabd21-de32-4922-b790-135ec968ff65\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"
Sep 15 22:00:47.468218 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.468182    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7kt7\" (UniqueName: \"kubernetes.io/projected/e3eabd21-de32-4922-b790-135ec968ff65-kube-api-access-x7kt7\") pod \"payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p\" (UID: \"e3eabd21-de32-4922-b790-135ec968ff65\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"
Sep 15 22:00:47.599875 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.599842    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"
Sep 15 22:00:47.883850 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:47.883821    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"]
Sep 15 22:00:47.885269 ip-10-0-135-148 kubenswrapper[2575]: W0915 22:00:47.885239    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3eabd21_de32_4922_b790_135ec968ff65.slice/crio-1cabf7486c8ea5e34eca356ac20397cc760e58722242f7d66bdb340b09a71575 WatchSource:0}: Error finding container 1cabf7486c8ea5e34eca356ac20397cc760e58722242f7d66bdb340b09a71575: Status 404 returned error can't find the container with id 1cabf7486c8ea5e34eca356ac20397cc760e58722242f7d66bdb340b09a71575
Sep 15 22:00:48.465799 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:48.465737    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p" event={"ID":"e3eabd21-de32-4922-b790-135ec968ff65","Type":"ContainerStarted","Data":"b8193e99db579a8f81bf9e6a96dfaf9082873199d7cb8e8483079836651041c4"}
Sep 15 22:00:48.466211 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:48.465829    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p" event={"ID":"e3eabd21-de32-4922-b790-135ec968ff65","Type":"ContainerStarted","Data":"1cabf7486c8ea5e34eca356ac20397cc760e58722242f7d66bdb340b09a71575"}
Sep 15 22:00:48.466211 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:48.465960    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"
Sep 15 22:00:48.494900 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:48.494843    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p" podStartSLOduration=1.494825323 podStartE2EDuration="1.494825323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:00:48.494716282 +0000 UTC m=+2086.306724728" watchObservedRunningTime="2026-09-15 22:00:48.494825323 +0000 UTC m=+2086.306833766"
Sep 15 22:00:52.246546 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:52.246508    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:00:59.477945 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:00:59.477907    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"
Sep 15 22:01:03.987656 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:03.987625    2575 scope.go:117] "RemoveContainer" containerID="21be41339e605d1a89ad12f184383977cc201aadcc65cbf506a4f87461ec8e6c"
Sep 15 22:01:03.996558 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:03.996532    2575 scope.go:117] "RemoveContainer" containerID="90eaac38faa7150e16477061b36baec66953080c5ef0fdecee4315332a61314b"
Sep 15 22:01:11.973239 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:11.973205    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt"]
Sep 15 22:01:11.976842 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:11.976826    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt"
Sep 15 22:01:11.980906 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:11.980886    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 22:01:11.980906 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:11.980900    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:01:11.997988 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:11.997968    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f725b70-d02f-4fef-b79f-ab026b3afbe8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt\" (UID: \"0f725b70-d02f-4fef-b79f-ab026b3afbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt"
Sep 15 22:01:12.004650 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:12.004621    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt"]
Sep 15 22:01:12.099352 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:12.099308    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f725b70-d02f-4fef-b79f-ab026b3afbe8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt\" (UID: \"0f725b70-d02f-4fef-b79f-ab026b3afbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt"
Sep 15 22:01:12.099989 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:12.099970    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f725b70-d02f-4fef-b79f-ab026b3afbe8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt\" (UID: \"0f725b70-d02f-4fef-b79f-ab026b3afbe8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt"
Sep 15 22:01:12.287603 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:12.287518    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt"
Sep 15 22:01:12.566519 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:12.566490    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt"]
Sep 15 22:01:12.567480 ip-10-0-135-148 kubenswrapper[2575]: W0915 22:01:12.567457    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f725b70_d02f_4fef_b79f_ab026b3afbe8.slice/crio-3dbeb7eb44a054cf191fb6bf13e7eb01c2f2c694893c95b2ce206dc86c7dbfd4 WatchSource:0}: Error finding container 3dbeb7eb44a054cf191fb6bf13e7eb01c2f2c694893c95b2ce206dc86c7dbfd4: Status 404 returned error can't find the container with id 3dbeb7eb44a054cf191fb6bf13e7eb01c2f2c694893c95b2ce206dc86c7dbfd4
Sep 15 22:01:12.598511 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:12.598484    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt" event={"ID":"0f725b70-d02f-4fef-b79f-ab026b3afbe8","Type":"ContainerStarted","Data":"3dbeb7eb44a054cf191fb6bf13e7eb01c2f2c694893c95b2ce206dc86c7dbfd4"}
Sep 15 22:01:13.605817 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:13.605787    2575 generic.go:356] "Generic (PLEG): container finished" podID="0f725b70-d02f-4fef-b79f-ab026b3afbe8" containerID="844b44f41a738654367d36a3f90adbb0f7e2ad8f248cf565e282c12577a57812" exitCode=0
Sep 15 22:01:13.606217 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:13.605874    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt" event={"ID":"0f725b70-d02f-4fef-b79f-ab026b3afbe8","Type":"ContainerDied","Data":"844b44f41a738654367d36a3f90adbb0f7e2ad8f248cf565e282c12577a57812"}
Sep 15 22:01:14.746395 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:14.746368    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt"
Sep 15 22:01:14.826958 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:14.826925    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f725b70-d02f-4fef-b79f-ab026b3afbe8-maas-api-service-ca\") pod \"0f725b70-d02f-4fef-b79f-ab026b3afbe8\" (UID: \"0f725b70-d02f-4fef-b79f-ab026b3afbe8\") "
Sep 15 22:01:14.827278 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:14.827256    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f725b70-d02f-4fef-b79f-ab026b3afbe8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f725b70-d02f-4fef-b79f-ab026b3afbe8" (UID: "0f725b70-d02f-4fef-b79f-ab026b3afbe8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:01:14.928082 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:14.927990    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f725b70-d02f-4fef-b79f-ab026b3afbe8-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:01:15.617502 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:15.617471    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt"
Sep 15 22:01:15.617703 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:15.617468    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt" event={"ID":"0f725b70-d02f-4fef-b79f-ab026b3afbe8","Type":"ContainerDied","Data":"3dbeb7eb44a054cf191fb6bf13e7eb01c2f2c694893c95b2ce206dc86c7dbfd4"}
Sep 15 22:01:15.617703 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:15.617600    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3dbeb7eb44a054cf191fb6bf13e7eb01c2f2c694893c95b2ce206dc86c7dbfd4"
Sep 15 22:01:24.115191 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.115161    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"]
Sep 15 22:01:24.115747 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.115350    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p" podUID="e3eabd21-de32-4922-b790-135ec968ff65" containerName="payload-processing" containerID="cri-o://b8193e99db579a8f81bf9e6a96dfaf9082873199d7cb8e8483079836651041c4" gracePeriod=30
Sep 15 22:01:24.368372 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.368297    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"
Sep 15 22:01:24.422461 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.422428    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7kt7\" (UniqueName: \"kubernetes.io/projected/e3eabd21-de32-4922-b790-135ec968ff65-kube-api-access-x7kt7\") pod \"e3eabd21-de32-4922-b790-135ec968ff65\" (UID: \"e3eabd21-de32-4922-b790-135ec968ff65\") "
Sep 15 22:01:24.422645 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.422527    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3eabd21-de32-4922-b790-135ec968ff65-plugins-config-volume\") pod \"e3eabd21-de32-4922-b790-135ec968ff65\" (UID: \"e3eabd21-de32-4922-b790-135ec968ff65\") "
Sep 15 22:01:24.422913 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.422889    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3eabd21-de32-4922-b790-135ec968ff65-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e3eabd21-de32-4922-b790-135ec968ff65" (UID: "e3eabd21-de32-4922-b790-135ec968ff65"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:01:24.424714 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.424694    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3eabd21-de32-4922-b790-135ec968ff65-kube-api-access-x7kt7" (OuterVolumeSpecName: "kube-api-access-x7kt7") pod "e3eabd21-de32-4922-b790-135ec968ff65" (UID: "e3eabd21-de32-4922-b790-135ec968ff65"). InnerVolumeSpecName "kube-api-access-x7kt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:01:24.460848 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.460817    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt"]
Sep 15 22:01:24.468694 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.468665    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3596d6-fmtjt"]
Sep 15 22:01:24.523958 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.523926    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3eabd21-de32-4922-b790-135ec968ff65-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:01:24.523958 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.523952    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7kt7\" (UniqueName: \"kubernetes.io/projected/e3eabd21-de32-4922-b790-135ec968ff65-kube-api-access-x7kt7\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:01:24.667562 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.667469    2575 generic.go:356] "Generic (PLEG): container finished" podID="e3eabd21-de32-4922-b790-135ec968ff65" containerID="b8193e99db579a8f81bf9e6a96dfaf9082873199d7cb8e8483079836651041c4" exitCode=0
Sep 15 22:01:24.667562 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.667513    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"
Sep 15 22:01:24.667562 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.667551    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p" event={"ID":"e3eabd21-de32-4922-b790-135ec968ff65","Type":"ContainerDied","Data":"b8193e99db579a8f81bf9e6a96dfaf9082873199d7cb8e8483079836651041c4"}
Sep 15 22:01:24.667870 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.667625    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p" event={"ID":"e3eabd21-de32-4922-b790-135ec968ff65","Type":"ContainerDied","Data":"1cabf7486c8ea5e34eca356ac20397cc760e58722242f7d66bdb340b09a71575"}
Sep 15 22:01:24.667870 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.667648    2575 scope.go:117] "RemoveContainer" containerID="b8193e99db579a8f81bf9e6a96dfaf9082873199d7cb8e8483079836651041c4"
Sep 15 22:01:24.677065 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.677049    2575 scope.go:117] "RemoveContainer" containerID="b8193e99db579a8f81bf9e6a96dfaf9082873199d7cb8e8483079836651041c4"
Sep 15 22:01:24.677324 ip-10-0-135-148 kubenswrapper[2575]: E0915 22:01:24.677305    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8193e99db579a8f81bf9e6a96dfaf9082873199d7cb8e8483079836651041c4\": container with ID starting with b8193e99db579a8f81bf9e6a96dfaf9082873199d7cb8e8483079836651041c4 not found: ID does not exist" containerID="b8193e99db579a8f81bf9e6a96dfaf9082873199d7cb8e8483079836651041c4"
Sep 15 22:01:24.677378 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.677332    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8193e99db579a8f81bf9e6a96dfaf9082873199d7cb8e8483079836651041c4"} err="failed to get container status \"b8193e99db579a8f81bf9e6a96dfaf9082873199d7cb8e8483079836651041c4\": rpc error: code = NotFound desc = could not find container \"b8193e99db579a8f81bf9e6a96dfaf9082873199d7cb8e8483079836651041c4\": container with ID starting with b8193e99db579a8f81bf9e6a96dfaf9082873199d7cb8e8483079836651041c4 not found: ID does not exist"
Sep 15 22:01:24.712678 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.712648    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"]
Sep 15 22:01:24.729378 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.729348    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3596d6-5f449cb5cc-2z97p"]
Sep 15 22:01:24.747983 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.747960    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f725b70-d02f-4fef-b79f-ab026b3afbe8" path="/var/lib/kubelet/pods/0f725b70-d02f-4fef-b79f-ab026b3afbe8/volumes"
Sep 15 22:01:24.748315 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:24.748303    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3eabd21-de32-4922-b790-135ec968ff65" path="/var/lib/kubelet/pods/e3eabd21-de32-4922-b790-135ec968ff65/volumes"
Sep 15 22:01:29.040272 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:29.040239    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:01:37.408076 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:37.408040    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm"]
Sep 15 22:01:37.408509 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:37.408496    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f725b70-d02f-4fef-b79f-ab026b3afbe8" containerName="revoke-keys"
Sep 15 22:01:37.408560 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:37.408511    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f725b70-d02f-4fef-b79f-ab026b3afbe8" containerName="revoke-keys"
Sep 15 22:01:37.408560 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:37.408530    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3eabd21-de32-4922-b790-135ec968ff65" containerName="payload-processing"
Sep 15 22:01:37.408560 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:37.408535    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3eabd21-de32-4922-b790-135ec968ff65" containerName="payload-processing"
Sep 15 22:01:37.408654 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:37.408594    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3eabd21-de32-4922-b790-135ec968ff65" containerName="payload-processing"
Sep 15 22:01:37.408654 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:37.408604    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f725b70-d02f-4fef-b79f-ab026b3afbe8" containerName="revoke-keys"
Sep 15 22:01:37.411637 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:37.411622    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm"
Sep 15 22:01:37.417244 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:37.417218    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 22:01:37.417408 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:37.417250    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:01:37.447197 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:37.447170    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm"]
Sep 15 22:01:37.556533 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:37.556501    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b02d2-abdc-45ff-a7dd-7a138c7da737-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm\" (UID: \"e77b02d2-abdc-45ff-a7dd-7a138c7da737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm"
Sep 15 22:01:37.657338 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:37.657306    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b02d2-abdc-45ff-a7dd-7a138c7da737-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm\" (UID: \"e77b02d2-abdc-45ff-a7dd-7a138c7da737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm"
Sep 15 22:01:37.657988 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:37.657963    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b02d2-abdc-45ff-a7dd-7a138c7da737-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm\" (UID: \"e77b02d2-abdc-45ff-a7dd-7a138c7da737\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm"
Sep 15 22:01:37.722497 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:37.722421    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm"
Sep 15 22:01:38.006358 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:38.006274    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm"]
Sep 15 22:01:38.011189 ip-10-0-135-148 kubenswrapper[2575]: W0915 22:01:38.011162    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77b02d2_abdc_45ff_a7dd_7a138c7da737.slice/crio-766a8ae9ac12a3faeec4a8a8f5e79a5ba1e488885e304fc630ef017edf0b074d WatchSource:0}: Error finding container 766a8ae9ac12a3faeec4a8a8f5e79a5ba1e488885e304fc630ef017edf0b074d: Status 404 returned error can't find the container with id 766a8ae9ac12a3faeec4a8a8f5e79a5ba1e488885e304fc630ef017edf0b074d
Sep 15 22:01:38.745594 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:38.745557    2575 generic.go:356] "Generic (PLEG): container finished" podID="e77b02d2-abdc-45ff-a7dd-7a138c7da737" containerID="ade3c345ca7aa04c8fa805c9c13600cfaf249a1db0db850c63a52106fe342ccb" exitCode=0
Sep 15 22:01:38.747814 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:38.747789    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm" event={"ID":"e77b02d2-abdc-45ff-a7dd-7a138c7da737","Type":"ContainerDied","Data":"ade3c345ca7aa04c8fa805c9c13600cfaf249a1db0db850c63a52106fe342ccb"}
Sep 15 22:01:38.747934 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:38.747834    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm" event={"ID":"e77b02d2-abdc-45ff-a7dd-7a138c7da737","Type":"ContainerStarted","Data":"766a8ae9ac12a3faeec4a8a8f5e79a5ba1e488885e304fc630ef017edf0b074d"}
Sep 15 22:01:39.889227 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:39.889205    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm"
Sep 15 22:01:40.082988 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:40.082901    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b02d2-abdc-45ff-a7dd-7a138c7da737-maas-api-service-ca\") pod \"e77b02d2-abdc-45ff-a7dd-7a138c7da737\" (UID: \"e77b02d2-abdc-45ff-a7dd-7a138c7da737\") "
Sep 15 22:01:40.083249 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:40.083228    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77b02d2-abdc-45ff-a7dd-7a138c7da737-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e77b02d2-abdc-45ff-a7dd-7a138c7da737" (UID: "e77b02d2-abdc-45ff-a7dd-7a138c7da737"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:01:40.184350 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:40.184316    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b02d2-abdc-45ff-a7dd-7a138c7da737-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:01:40.759952 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:40.759924    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm" event={"ID":"e77b02d2-abdc-45ff-a7dd-7a138c7da737","Type":"ContainerDied","Data":"766a8ae9ac12a3faeec4a8a8f5e79a5ba1e488885e304fc630ef017edf0b074d"}
Sep 15 22:01:40.760089 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:40.759971    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="766a8ae9ac12a3faeec4a8a8f5e79a5ba1e488885e304fc630ef017edf0b074d"
Sep 15 22:01:40.760089 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:40.759936    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm"
Sep 15 22:01:51.050384 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:51.050350    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:01:53.486526 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.486438    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"]
Sep 15 22:01:53.486960 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.486696    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng" podUID="44877669-864e-44ff-aefd-9c6344f85d1c" containerName="payload-processing" containerID="cri-o://0c630e1f5966a5e5765988a42f8a467d5e455caf89d2ffffc7f169eef1e91d0f" gracePeriod=30
Sep 15 22:01:53.512045 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.512010    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"]
Sep 15 22:01:53.512297 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.512245    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879" podUID="ac3b05cb-614a-41c9-8764-53091ff06c83" containerName="payload-pre-processing" containerID="cri-o://d9be505e40c8dec62e6be25660e39b459e1b1b802900bb441c3b16e7f957ccb4" gracePeriod=30
Sep 15 22:01:53.772466 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.772439    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"
Sep 15 22:01:53.799948 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.799920    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"
Sep 15 22:01:53.807881 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.807854    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm"]
Sep 15 22:01:53.809869 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.809852    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4r9f\" (UniqueName: \"kubernetes.io/projected/44877669-864e-44ff-aefd-9c6344f85d1c-kube-api-access-l4r9f\") pod \"44877669-864e-44ff-aefd-9c6344f85d1c\" (UID: \"44877669-864e-44ff-aefd-9c6344f85d1c\") "
Sep 15 22:01:53.809948 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.809937    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44877669-864e-44ff-aefd-9c6344f85d1c-plugins-config-volume\") pod \"44877669-864e-44ff-aefd-9c6344f85d1c\" (UID: \"44877669-864e-44ff-aefd-9c6344f85d1c\") "
Sep 15 22:01:53.812692 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.811717    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44877669-864e-44ff-aefd-9c6344f85d1c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44877669-864e-44ff-aefd-9c6344f85d1c" (UID: "44877669-864e-44ff-aefd-9c6344f85d1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:01:53.815810 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.813298    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44877669-864e-44ff-aefd-9c6344f85d1c-kube-api-access-l4r9f" (OuterVolumeSpecName: "kube-api-access-l4r9f") pod "44877669-864e-44ff-aefd-9c6344f85d1c" (UID: "44877669-864e-44ff-aefd-9c6344f85d1c"). InnerVolumeSpecName "kube-api-access-l4r9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:01:53.815810 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.815559    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44877669-864e-44ff-aefd-9c6344f85d1c-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:01:53.817456 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.817425    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-0886df-jz6xm"]
Sep 15 22:01:53.834360 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.834322    2575 generic.go:356] "Generic (PLEG): container finished" podID="ac3b05cb-614a-41c9-8764-53091ff06c83" containerID="d9be505e40c8dec62e6be25660e39b459e1b1b802900bb441c3b16e7f957ccb4" exitCode=0
Sep 15 22:01:53.834535 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.834369    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"
Sep 15 22:01:53.834535 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.834405    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879" event={"ID":"ac3b05cb-614a-41c9-8764-53091ff06c83","Type":"ContainerDied","Data":"d9be505e40c8dec62e6be25660e39b459e1b1b802900bb441c3b16e7f957ccb4"}
Sep 15 22:01:53.834535 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.834480    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879" event={"ID":"ac3b05cb-614a-41c9-8764-53091ff06c83","Type":"ContainerDied","Data":"82ea1e51a4464c5c8a281b04407027d4932d4326fbcaafcfa16486abbfff570b"}
Sep 15 22:01:53.834535 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.834508    2575 scope.go:117] "RemoveContainer" containerID="d9be505e40c8dec62e6be25660e39b459e1b1b802900bb441c3b16e7f957ccb4"
Sep 15 22:01:53.836226 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.836199    2575 generic.go:356] "Generic (PLEG): container finished" podID="44877669-864e-44ff-aefd-9c6344f85d1c" containerID="0c630e1f5966a5e5765988a42f8a467d5e455caf89d2ffffc7f169eef1e91d0f" exitCode=0
Sep 15 22:01:53.836321 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.836249    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"
Sep 15 22:01:53.836321 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.836263    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng" event={"ID":"44877669-864e-44ff-aefd-9c6344f85d1c","Type":"ContainerDied","Data":"0c630e1f5966a5e5765988a42f8a467d5e455caf89d2ffffc7f169eef1e91d0f"}
Sep 15 22:01:53.836321 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.836300    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng" event={"ID":"44877669-864e-44ff-aefd-9c6344f85d1c","Type":"ContainerDied","Data":"c31982cb722029b39fd95150413a1068820ecc4f20a5a7e85db498844aff84d1"}
Sep 15 22:01:53.844246 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.844181    2575 scope.go:117] "RemoveContainer" containerID="d9be505e40c8dec62e6be25660e39b459e1b1b802900bb441c3b16e7f957ccb4"
Sep 15 22:01:53.844568 ip-10-0-135-148 kubenswrapper[2575]: E0915 22:01:53.844544    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9be505e40c8dec62e6be25660e39b459e1b1b802900bb441c3b16e7f957ccb4\": container with ID starting with d9be505e40c8dec62e6be25660e39b459e1b1b802900bb441c3b16e7f957ccb4 not found: ID does not exist" containerID="d9be505e40c8dec62e6be25660e39b459e1b1b802900bb441c3b16e7f957ccb4"
Sep 15 22:01:53.844650 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.844576    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9be505e40c8dec62e6be25660e39b459e1b1b802900bb441c3b16e7f957ccb4"} err="failed to get container status \"d9be505e40c8dec62e6be25660e39b459e1b1b802900bb441c3b16e7f957ccb4\": rpc error: code = NotFound desc = could not find container \"d9be505e40c8dec62e6be25660e39b459e1b1b802900bb441c3b16e7f957ccb4\": container with ID starting with d9be505e40c8dec62e6be25660e39b459e1b1b802900bb441c3b16e7f957ccb4 not found: ID does not exist"
Sep 15 22:01:53.844650 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.844593    2575 scope.go:117] "RemoveContainer" containerID="0c630e1f5966a5e5765988a42f8a467d5e455caf89d2ffffc7f169eef1e91d0f"
Sep 15 22:01:53.853862 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.853839    2575 scope.go:117] "RemoveContainer" containerID="0c630e1f5966a5e5765988a42f8a467d5e455caf89d2ffffc7f169eef1e91d0f"
Sep 15 22:01:53.854115 ip-10-0-135-148 kubenswrapper[2575]: E0915 22:01:53.854099    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c630e1f5966a5e5765988a42f8a467d5e455caf89d2ffffc7f169eef1e91d0f\": container with ID starting with 0c630e1f5966a5e5765988a42f8a467d5e455caf89d2ffffc7f169eef1e91d0f not found: ID does not exist" containerID="0c630e1f5966a5e5765988a42f8a467d5e455caf89d2ffffc7f169eef1e91d0f"
Sep 15 22:01:53.854177 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.854122    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c630e1f5966a5e5765988a42f8a467d5e455caf89d2ffffc7f169eef1e91d0f"} err="failed to get container status \"0c630e1f5966a5e5765988a42f8a467d5e455caf89d2ffffc7f169eef1e91d0f\": rpc error: code = NotFound desc = could not find container \"0c630e1f5966a5e5765988a42f8a467d5e455caf89d2ffffc7f169eef1e91d0f\": container with ID starting with 0c630e1f5966a5e5765988a42f8a467d5e455caf89d2ffffc7f169eef1e91d0f not found: ID does not exist"
Sep 15 22:01:53.865420 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.865379    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"]
Sep 15 22:01:53.870217 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.870185    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0886df-79d59cf49-wjnng"]
Sep 15 22:01:53.916588 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.916557    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac3b05cb-614a-41c9-8764-53091ff06c83-plugins-config-volume\") pod \"ac3b05cb-614a-41c9-8764-53091ff06c83\" (UID: \"ac3b05cb-614a-41c9-8764-53091ff06c83\") "
Sep 15 22:01:53.916791 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.916597    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4njhf\" (UniqueName: \"kubernetes.io/projected/ac3b05cb-614a-41c9-8764-53091ff06c83-kube-api-access-4njhf\") pod \"ac3b05cb-614a-41c9-8764-53091ff06c83\" (UID: \"ac3b05cb-614a-41c9-8764-53091ff06c83\") "
Sep 15 22:01:53.916860 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.916846    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4r9f\" (UniqueName: \"kubernetes.io/projected/44877669-864e-44ff-aefd-9c6344f85d1c-kube-api-access-l4r9f\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:01:53.916995 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.916974    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3b05cb-614a-41c9-8764-53091ff06c83-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac3b05cb-614a-41c9-8764-53091ff06c83" (UID: "ac3b05cb-614a-41c9-8764-53091ff06c83"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:01:53.918954 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:53.918932    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3b05cb-614a-41c9-8764-53091ff06c83-kube-api-access-4njhf" (OuterVolumeSpecName: "kube-api-access-4njhf") pod "ac3b05cb-614a-41c9-8764-53091ff06c83" (UID: "ac3b05cb-614a-41c9-8764-53091ff06c83"). InnerVolumeSpecName "kube-api-access-4njhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:01:54.017594 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:54.017508    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac3b05cb-614a-41c9-8764-53091ff06c83-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:01:54.017594 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:54.017541    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4njhf\" (UniqueName: \"kubernetes.io/projected/ac3b05cb-614a-41c9-8764-53091ff06c83-kube-api-access-4njhf\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:01:54.164387 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:54.164353    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"]
Sep 15 22:01:54.168453 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:54.168428    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0886df-794cb57d7d-gk879"]
Sep 15 22:01:54.747847 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:54.747814    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44877669-864e-44ff-aefd-9c6344f85d1c" path="/var/lib/kubelet/pods/44877669-864e-44ff-aefd-9c6344f85d1c/volumes"
Sep 15 22:01:54.748212 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:54.748190    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac3b05cb-614a-41c9-8764-53091ff06c83" path="/var/lib/kubelet/pods/ac3b05cb-614a-41c9-8764-53091ff06c83/volumes"
Sep 15 22:01:54.748482 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:01:54.748469    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e77b02d2-abdc-45ff-a7dd-7a138c7da737" path="/var/lib/kubelet/pods/e77b02d2-abdc-45ff-a7dd-7a138c7da737/volumes"
Sep 15 22:02:04.020231 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:04.020198    2575 scope.go:117] "RemoveContainer" containerID="4b8720e01c42e3b18acbed79e68df7967ac31c0cd243f7e19b04301b5d82ff1f"
Sep 15 22:02:04.269179 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:04.269145    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:02:57.627826 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:57.627793    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"]
Sep 15 22:02:57.628363 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:57.628029    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp" podUID="42beaef4-865f-4831-8a1d-30fc885724ff" containerName="istio-proxy" containerID="cri-o://3885d24171d6c3acaf822823e390984e1df841741158730f2fa392b67adbf7cd" gracePeriod=30
Sep 15 22:02:58.005128 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.005039    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc"]
Sep 15 22:02:58.005545 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.005531    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44877669-864e-44ff-aefd-9c6344f85d1c" containerName="payload-processing"
Sep 15 22:02:58.005593 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.005551    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="44877669-864e-44ff-aefd-9c6344f85d1c" containerName="payload-processing"
Sep 15 22:02:58.005593 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.005567    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac3b05cb-614a-41c9-8764-53091ff06c83" containerName="payload-pre-processing"
Sep 15 22:02:58.005593 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.005572    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac3b05cb-614a-41c9-8764-53091ff06c83" containerName="payload-pre-processing"
Sep 15 22:02:58.005593 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.005582    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e77b02d2-abdc-45ff-a7dd-7a138c7da737" containerName="revoke-keys"
Sep 15 22:02:58.005593 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.005588    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e77b02d2-abdc-45ff-a7dd-7a138c7da737" containerName="revoke-keys"
Sep 15 22:02:58.005776 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.005658    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac3b05cb-614a-41c9-8764-53091ff06c83" containerName="payload-pre-processing"
Sep 15 22:02:58.005776 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.005666    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e77b02d2-abdc-45ff-a7dd-7a138c7da737" containerName="revoke-keys"
Sep 15 22:02:58.005776 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.005679    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="44877669-864e-44ff-aefd-9c6344f85d1c" containerName="payload-processing"
Sep 15 22:02:58.008935 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.008917    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc"
Sep 15 22:02:58.013814 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.013793    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 22:02:58.013942 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.013877    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:02:58.017920 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.017893    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc"]
Sep 15 22:02:58.139468 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.139431    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94222d30-21e4-42ea-8aee-74c616915273-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc\" (UID: \"94222d30-21e4-42ea-8aee-74c616915273\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc"
Sep 15 22:02:58.241394 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.240428    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94222d30-21e4-42ea-8aee-74c616915273-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc\" (UID: \"94222d30-21e4-42ea-8aee-74c616915273\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc"
Sep 15 22:02:58.241394 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.241344    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94222d30-21e4-42ea-8aee-74c616915273-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc\" (UID: \"94222d30-21e4-42ea-8aee-74c616915273\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc"
Sep 15 22:02:58.321058 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.320972    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc"
Sep 15 22:02:58.601074 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:58.601051    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc"]
Sep 15 22:02:58.603692 ip-10-0-135-148 kubenswrapper[2575]: W0915 22:02:58.603659    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94222d30_21e4_42ea_8aee_74c616915273.slice/crio-c89adde52495e3ccd8aabf0bf203aee5a42c5ebf01851df8cd9211d27b57d54f WatchSource:0}: Error finding container c89adde52495e3ccd8aabf0bf203aee5a42c5ebf01851df8cd9211d27b57d54f: Status 404 returned error can't find the container with id c89adde52495e3ccd8aabf0bf203aee5a42c5ebf01851df8cd9211d27b57d54f
Sep 15 22:02:59.182344 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:59.182254    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc" event={"ID":"94222d30-21e4-42ea-8aee-74c616915273","Type":"ContainerStarted","Data":"8f77ecfe57ac562709f06b5f47577861a7d7e7d1ce7abf5df3da65711f7386ac"}
Sep 15 22:02:59.182344 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:59.182315    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc" event={"ID":"94222d30-21e4-42ea-8aee-74c616915273","Type":"ContainerStarted","Data":"c89adde52495e3ccd8aabf0bf203aee5a42c5ebf01851df8cd9211d27b57d54f"}
Sep 15 22:02:59.200771 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:02:59.200705    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc" podStartSLOduration=2.200689158 podStartE2EDuration="2.200689158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:02:59.198795734 +0000 UTC m=+2217.010804181" watchObservedRunningTime="2026-09-15 22:02:59.200689158 +0000 UTC m=+2217.012697601"
Sep 15 22:03:00.192443 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:00.192412    2575 generic.go:356] "Generic (PLEG): container finished" podID="94222d30-21e4-42ea-8aee-74c616915273" containerID="8f77ecfe57ac562709f06b5f47577861a7d7e7d1ce7abf5df3da65711f7386ac" exitCode=0
Sep 15 22:03:00.192852 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:00.192503    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc" event={"ID":"94222d30-21e4-42ea-8aee-74c616915273","Type":"ContainerDied","Data":"8f77ecfe57ac562709f06b5f47577861a7d7e7d1ce7abf5df3da65711f7386ac"}
Sep 15 22:03:01.332005 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:01.331979    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc"
Sep 15 22:03:01.369047 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:01.369012    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94222d30-21e4-42ea-8aee-74c616915273-maas-api-service-ca\") pod \"94222d30-21e4-42ea-8aee-74c616915273\" (UID: \"94222d30-21e4-42ea-8aee-74c616915273\") "
Sep 15 22:03:01.369375 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:01.369354    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94222d30-21e4-42ea-8aee-74c616915273-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94222d30-21e4-42ea-8aee-74c616915273" (UID: "94222d30-21e4-42ea-8aee-74c616915273"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:03:01.369468 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:01.369456    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94222d30-21e4-42ea-8aee-74c616915273-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:02.205024 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.204997    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc"
Sep 15 22:03:02.205176 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.204993    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc" event={"ID":"94222d30-21e4-42ea-8aee-74c616915273","Type":"ContainerDied","Data":"c89adde52495e3ccd8aabf0bf203aee5a42c5ebf01851df8cd9211d27b57d54f"}
Sep 15 22:03:02.205176 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.205116    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c89adde52495e3ccd8aabf0bf203aee5a42c5ebf01851df8cd9211d27b57d54f"
Sep 15 22:03:02.877884 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.877860    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 22:03:02.886493 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.886468    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42beaef4-865f-4831-8a1d-30fc885724ff-istio-podinfo\") pod \"42beaef4-865f-4831-8a1d-30fc885724ff\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") "
Sep 15 22:03:02.886631 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.886502    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-credential-socket\") pod \"42beaef4-865f-4831-8a1d-30fc885724ff\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") "
Sep 15 22:03:02.886631 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.886545    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whwtz\" (UniqueName: \"kubernetes.io/projected/42beaef4-865f-4831-8a1d-30fc885724ff-kube-api-access-whwtz\") pod \"42beaef4-865f-4831-8a1d-30fc885724ff\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") "
Sep 15 22:03:02.886631 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.886590    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-workload-socket\") pod \"42beaef4-865f-4831-8a1d-30fc885724ff\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") "
Sep 15 22:03:02.886821 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.886632    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42beaef4-865f-4831-8a1d-30fc885724ff-istio-token\") pod \"42beaef4-865f-4831-8a1d-30fc885724ff\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") "
Sep 15 22:03:02.886821 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.886683    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-istio-envoy\") pod \"42beaef4-865f-4831-8a1d-30fc885724ff\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") "
Sep 15 22:03:02.886821 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.886716    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-istio-data\") pod \"42beaef4-865f-4831-8a1d-30fc885724ff\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") "
Sep 15 22:03:02.886821 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.886735    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "42beaef4-865f-4831-8a1d-30fc885724ff" (UID: "42beaef4-865f-4831-8a1d-30fc885724ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:03:02.886821 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.886799    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-workload-certs\") pod \"42beaef4-865f-4831-8a1d-30fc885724ff\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") "
Sep 15 22:03:02.887071 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.886835    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42beaef4-865f-4831-8a1d-30fc885724ff-istiod-ca-cert\") pod \"42beaef4-865f-4831-8a1d-30fc885724ff\" (UID: \"42beaef4-865f-4831-8a1d-30fc885724ff\") "
Sep 15 22:03:02.887071 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.886930    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "42beaef4-865f-4831-8a1d-30fc885724ff" (UID: "42beaef4-865f-4831-8a1d-30fc885724ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:03:02.887071 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.887009    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-istio-data" (OuterVolumeSpecName: "istio-data") pod "42beaef4-865f-4831-8a1d-30fc885724ff" (UID: "42beaef4-865f-4831-8a1d-30fc885724ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:03:02.887204 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.887144    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "42beaef4-865f-4831-8a1d-30fc885724ff" (UID: "42beaef4-865f-4831-8a1d-30fc885724ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:03:02.887448 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.887336    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42beaef4-865f-4831-8a1d-30fc885724ff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "42beaef4-865f-4831-8a1d-30fc885724ff" (UID: "42beaef4-865f-4831-8a1d-30fc885724ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:03:02.887538 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.887519    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:02.887579 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.887546    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:02.887579 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.887560    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:02.887579 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.887573    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:02.888882 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.888859    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42beaef4-865f-4831-8a1d-30fc885724ff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "42beaef4-865f-4831-8a1d-30fc885724ff" (UID: "42beaef4-865f-4831-8a1d-30fc885724ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 22:03:02.888992 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.888976    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "42beaef4-865f-4831-8a1d-30fc885724ff" (UID: "42beaef4-865f-4831-8a1d-30fc885724ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:03:02.889246 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.889224    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42beaef4-865f-4831-8a1d-30fc885724ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "42beaef4-865f-4831-8a1d-30fc885724ff" (UID: "42beaef4-865f-4831-8a1d-30fc885724ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:03:02.889320 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.889246    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42beaef4-865f-4831-8a1d-30fc885724ff-kube-api-access-whwtz" (OuterVolumeSpecName: "kube-api-access-whwtz") pod "42beaef4-865f-4831-8a1d-30fc885724ff" (UID: "42beaef4-865f-4831-8a1d-30fc885724ff"). InnerVolumeSpecName "kube-api-access-whwtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:03:02.988320 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.988284    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42beaef4-865f-4831-8a1d-30fc885724ff-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:02.988320 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.988314    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42beaef4-865f-4831-8a1d-30fc885724ff-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:02.988320 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.988323    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42beaef4-865f-4831-8a1d-30fc885724ff-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:02.988570 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.988334    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whwtz\" (UniqueName: \"kubernetes.io/projected/42beaef4-865f-4831-8a1d-30fc885724ff-kube-api-access-whwtz\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:02.988570 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:02.988343    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42beaef4-865f-4831-8a1d-30fc885724ff-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:03.211745 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:03.211656    2575 generic.go:356] "Generic (PLEG): container finished" podID="42beaef4-865f-4831-8a1d-30fc885724ff" containerID="3885d24171d6c3acaf822823e390984e1df841741158730f2fa392b67adbf7cd" exitCode=0
Sep 15 22:03:03.211745 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:03.211702    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"
Sep 15 22:03:03.211934 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:03.211745    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp" event={"ID":"42beaef4-865f-4831-8a1d-30fc885724ff","Type":"ContainerDied","Data":"3885d24171d6c3acaf822823e390984e1df841741158730f2fa392b67adbf7cd"}
Sep 15 22:03:03.211934 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:03.211824    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp" event={"ID":"42beaef4-865f-4831-8a1d-30fc885724ff","Type":"ContainerDied","Data":"d554d2638585fc0e0c13dfe30254c63735c5854eb8dc9f953e5355814751ed04"}
Sep 15 22:03:03.211934 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:03.211858    2575 scope.go:117] "RemoveContainer" containerID="3885d24171d6c3acaf822823e390984e1df841741158730f2fa392b67adbf7cd"
Sep 15 22:03:03.222092 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:03.222076    2575 scope.go:117] "RemoveContainer" containerID="4b94ea6ae910c0bb89f00b2cfd59b93f05ba21d364091f8ef72351b320ed5480"
Sep 15 22:03:03.230301 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:03.230286    2575 scope.go:117] "RemoveContainer" containerID="3885d24171d6c3acaf822823e390984e1df841741158730f2fa392b67adbf7cd"
Sep 15 22:03:03.230553 ip-10-0-135-148 kubenswrapper[2575]: E0915 22:03:03.230536    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3885d24171d6c3acaf822823e390984e1df841741158730f2fa392b67adbf7cd\": container with ID starting with 3885d24171d6c3acaf822823e390984e1df841741158730f2fa392b67adbf7cd not found: ID does not exist" containerID="3885d24171d6c3acaf822823e390984e1df841741158730f2fa392b67adbf7cd"
Sep 15 22:03:03.230606 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:03.230561    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3885d24171d6c3acaf822823e390984e1df841741158730f2fa392b67adbf7cd"} err="failed to get container status \"3885d24171d6c3acaf822823e390984e1df841741158730f2fa392b67adbf7cd\": rpc error: code = NotFound desc = could not find container \"3885d24171d6c3acaf822823e390984e1df841741158730f2fa392b67adbf7cd\": container with ID starting with 3885d24171d6c3acaf822823e390984e1df841741158730f2fa392b67adbf7cd not found: ID does not exist"
Sep 15 22:03:03.230606 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:03.230577    2575 scope.go:117] "RemoveContainer" containerID="4b94ea6ae910c0bb89f00b2cfd59b93f05ba21d364091f8ef72351b320ed5480"
Sep 15 22:03:03.230837 ip-10-0-135-148 kubenswrapper[2575]: E0915 22:03:03.230819    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b94ea6ae910c0bb89f00b2cfd59b93f05ba21d364091f8ef72351b320ed5480\": container with ID starting with 4b94ea6ae910c0bb89f00b2cfd59b93f05ba21d364091f8ef72351b320ed5480 not found: ID does not exist" containerID="4b94ea6ae910c0bb89f00b2cfd59b93f05ba21d364091f8ef72351b320ed5480"
Sep 15 22:03:03.230877 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:03.230843    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b94ea6ae910c0bb89f00b2cfd59b93f05ba21d364091f8ef72351b320ed5480"} err="failed to get container status \"4b94ea6ae910c0bb89f00b2cfd59b93f05ba21d364091f8ef72351b320ed5480\": rpc error: code = NotFound desc = could not find container \"4b94ea6ae910c0bb89f00b2cfd59b93f05ba21d364091f8ef72351b320ed5480\": container with ID starting with 4b94ea6ae910c0bb89f00b2cfd59b93f05ba21d364091f8ef72351b320ed5480 not found: ID does not exist"
Sep 15 22:03:03.237474 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:03.237442    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"]
Sep 15 22:03:03.243177 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:03.243150    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-0886df-openshift-default-b5ffc8f8-4cpdp"]
Sep 15 22:03:04.747460 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:04.747425    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42beaef4-865f-4831-8a1d-30fc885724ff" path="/var/lib/kubelet/pods/42beaef4-865f-4831-8a1d-30fc885724ff/volumes"
Sep 15 22:03:08.566693 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.566646    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1297a3-29825160-6g8jd"]
Sep 15 22:03:08.568799 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.568776    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-1297a3-29825160-6g8jd"]
Sep 15 22:03:08.592556 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.592528    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"]
Sep 15 22:03:08.592787 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.592740    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz" podUID="79c17a0d-49c8-4126-88a2-664e666d84d0" containerName="payload-processing" containerID="cri-o://e52c4c395ffda85565bb548625a0481b5d39a667054facc27a87b17ca57da657" gracePeriod=30
Sep 15 22:03:08.614897 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.611457    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"]
Sep 15 22:03:08.614897 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.611797    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl" podUID="17f095e5-a6e0-4fff-9ccb-36ded38233a1" containerName="payload-pre-processing" containerID="cri-o://8d3f556a107446c3ad774aa78b46fec01c47beeef4d2daa04a0420196a25a785" gracePeriod=30
Sep 15 22:03:08.747578 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.747544    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec3a1b68-1489-44eb-b26d-45cb64f4711e" path="/var/lib/kubelet/pods/ec3a1b68-1489-44eb-b26d-45cb64f4711e/volumes"
Sep 15 22:03:08.824233 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.824183    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc"]
Sep 15 22:03:08.829693 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.829666    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1297a3-xqhqc"]
Sep 15 22:03:08.893063 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.893035    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"
Sep 15 22:03:08.895657 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.895640    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"
Sep 15 22:03:08.946341 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.946298    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjd4v\" (UniqueName: \"kubernetes.io/projected/79c17a0d-49c8-4126-88a2-664e666d84d0-kube-api-access-xjd4v\") pod \"79c17a0d-49c8-4126-88a2-664e666d84d0\" (UID: \"79c17a0d-49c8-4126-88a2-664e666d84d0\") "
Sep 15 22:03:08.946534 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.946405    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c17a0d-49c8-4126-88a2-664e666d84d0-plugins-config-volume\") pod \"79c17a0d-49c8-4126-88a2-664e666d84d0\" (UID: \"79c17a0d-49c8-4126-88a2-664e666d84d0\") "
Sep 15 22:03:08.946534 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.946475    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f095e5-a6e0-4fff-9ccb-36ded38233a1-plugins-config-volume\") pod \"17f095e5-a6e0-4fff-9ccb-36ded38233a1\" (UID: \"17f095e5-a6e0-4fff-9ccb-36ded38233a1\") "
Sep 15 22:03:08.946534 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.946513    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpdll\" (UniqueName: \"kubernetes.io/projected/17f095e5-a6e0-4fff-9ccb-36ded38233a1-kube-api-access-rpdll\") pod \"17f095e5-a6e0-4fff-9ccb-36ded38233a1\" (UID: \"17f095e5-a6e0-4fff-9ccb-36ded38233a1\") "
Sep 15 22:03:08.946899 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.946872    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c17a0d-49c8-4126-88a2-664e666d84d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "79c17a0d-49c8-4126-88a2-664e666d84d0" (UID: "79c17a0d-49c8-4126-88a2-664e666d84d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:03:08.946971 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.946895    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f095e5-a6e0-4fff-9ccb-36ded38233a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17f095e5-a6e0-4fff-9ccb-36ded38233a1" (UID: "17f095e5-a6e0-4fff-9ccb-36ded38233a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:03:08.948931 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.948897    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f095e5-a6e0-4fff-9ccb-36ded38233a1-kube-api-access-rpdll" (OuterVolumeSpecName: "kube-api-access-rpdll") pod "17f095e5-a6e0-4fff-9ccb-36ded38233a1" (UID: "17f095e5-a6e0-4fff-9ccb-36ded38233a1"). InnerVolumeSpecName "kube-api-access-rpdll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:03:08.949095 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:08.948994    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c17a0d-49c8-4126-88a2-664e666d84d0-kube-api-access-xjd4v" (OuterVolumeSpecName: "kube-api-access-xjd4v") pod "79c17a0d-49c8-4126-88a2-664e666d84d0" (UID: "79c17a0d-49c8-4126-88a2-664e666d84d0"). InnerVolumeSpecName "kube-api-access-xjd4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:03:09.048291 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.048250    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79c17a0d-49c8-4126-88a2-664e666d84d0-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:09.048291 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.048283    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17f095e5-a6e0-4fff-9ccb-36ded38233a1-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:09.048291 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.048292    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpdll\" (UniqueName: \"kubernetes.io/projected/17f095e5-a6e0-4fff-9ccb-36ded38233a1-kube-api-access-rpdll\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:09.048291 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.048300    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjd4v\" (UniqueName: \"kubernetes.io/projected/79c17a0d-49c8-4126-88a2-664e666d84d0-kube-api-access-xjd4v\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:09.246574 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.246537    2575 generic.go:356] "Generic (PLEG): container finished" podID="17f095e5-a6e0-4fff-9ccb-36ded38233a1" containerID="8d3f556a107446c3ad774aa78b46fec01c47beeef4d2daa04a0420196a25a785" exitCode=0
Sep 15 22:03:09.246875 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.246582    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"
Sep 15 22:03:09.246875 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.246624    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl" event={"ID":"17f095e5-a6e0-4fff-9ccb-36ded38233a1","Type":"ContainerDied","Data":"8d3f556a107446c3ad774aa78b46fec01c47beeef4d2daa04a0420196a25a785"}
Sep 15 22:03:09.246875 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.246690    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl" event={"ID":"17f095e5-a6e0-4fff-9ccb-36ded38233a1","Type":"ContainerDied","Data":"556d1f946757b940fe07fdc5e7a128b9ba384f4567e8630d145c3070cc85cbf7"}
Sep 15 22:03:09.246875 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.246727    2575 scope.go:117] "RemoveContainer" containerID="8d3f556a107446c3ad774aa78b46fec01c47beeef4d2daa04a0420196a25a785"
Sep 15 22:03:09.248192 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.248167    2575 generic.go:356] "Generic (PLEG): container finished" podID="79c17a0d-49c8-4126-88a2-664e666d84d0" containerID="e52c4c395ffda85565bb548625a0481b5d39a667054facc27a87b17ca57da657" exitCode=0
Sep 15 22:03:09.248312 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.248196    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz" event={"ID":"79c17a0d-49c8-4126-88a2-664e666d84d0","Type":"ContainerDied","Data":"e52c4c395ffda85565bb548625a0481b5d39a667054facc27a87b17ca57da657"}
Sep 15 22:03:09.248312 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.248217    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"
Sep 15 22:03:09.248312 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.248237    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz" event={"ID":"79c17a0d-49c8-4126-88a2-664e666d84d0","Type":"ContainerDied","Data":"2e6254f17060ea64cc7f449bdbb929c8027d0ededd835d39b503268003069269"}
Sep 15 22:03:09.257977 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.257956    2575 scope.go:117] "RemoveContainer" containerID="8d3f556a107446c3ad774aa78b46fec01c47beeef4d2daa04a0420196a25a785"
Sep 15 22:03:09.258371 ip-10-0-135-148 kubenswrapper[2575]: E0915 22:03:09.258312    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d3f556a107446c3ad774aa78b46fec01c47beeef4d2daa04a0420196a25a785\": container with ID starting with 8d3f556a107446c3ad774aa78b46fec01c47beeef4d2daa04a0420196a25a785 not found: ID does not exist" containerID="8d3f556a107446c3ad774aa78b46fec01c47beeef4d2daa04a0420196a25a785"
Sep 15 22:03:09.258478 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.258382    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d3f556a107446c3ad774aa78b46fec01c47beeef4d2daa04a0420196a25a785"} err="failed to get container status \"8d3f556a107446c3ad774aa78b46fec01c47beeef4d2daa04a0420196a25a785\": rpc error: code = NotFound desc = could not find container \"8d3f556a107446c3ad774aa78b46fec01c47beeef4d2daa04a0420196a25a785\": container with ID starting with 8d3f556a107446c3ad774aa78b46fec01c47beeef4d2daa04a0420196a25a785 not found: ID does not exist"
Sep 15 22:03:09.258478 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.258407    2575 scope.go:117] "RemoveContainer" containerID="e52c4c395ffda85565bb548625a0481b5d39a667054facc27a87b17ca57da657"
Sep 15 22:03:09.267409 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.267389    2575 scope.go:117] "RemoveContainer" containerID="e52c4c395ffda85565bb548625a0481b5d39a667054facc27a87b17ca57da657"
Sep 15 22:03:09.267716 ip-10-0-135-148 kubenswrapper[2575]: E0915 22:03:09.267695    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52c4c395ffda85565bb548625a0481b5d39a667054facc27a87b17ca57da657\": container with ID starting with e52c4c395ffda85565bb548625a0481b5d39a667054facc27a87b17ca57da657 not found: ID does not exist" containerID="e52c4c395ffda85565bb548625a0481b5d39a667054facc27a87b17ca57da657"
Sep 15 22:03:09.267813 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.267725    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52c4c395ffda85565bb548625a0481b5d39a667054facc27a87b17ca57da657"} err="failed to get container status \"e52c4c395ffda85565bb548625a0481b5d39a667054facc27a87b17ca57da657\": rpc error: code = NotFound desc = could not find container \"e52c4c395ffda85565bb548625a0481b5d39a667054facc27a87b17ca57da657\": container with ID starting with e52c4c395ffda85565bb548625a0481b5d39a667054facc27a87b17ca57da657 not found: ID does not exist"
Sep 15 22:03:09.272784 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.272738    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"]
Sep 15 22:03:09.280857 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.280828    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1297a3-5d89f4cd-72jdl"]
Sep 15 22:03:09.297480 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.297452    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"]
Sep 15 22:03:09.303971 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:09.303947    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1297a3-6d666f8bb9-s6hsz"]
Sep 15 22:03:10.748215 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:10.748184    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17f095e5-a6e0-4fff-9ccb-36ded38233a1" path="/var/lib/kubelet/pods/17f095e5-a6e0-4fff-9ccb-36ded38233a1/volumes"
Sep 15 22:03:10.748598 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:10.748556    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79c17a0d-49c8-4126-88a2-664e666d84d0" path="/var/lib/kubelet/pods/79c17a0d-49c8-4126-88a2-664e666d84d0/volumes"
Sep 15 22:03:10.748858 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:10.748843    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94222d30-21e4-42ea-8aee-74c616915273" path="/var/lib/kubelet/pods/94222d30-21e4-42ea-8aee-74c616915273/volumes"
Sep 15 22:03:12.137982 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:12.137942    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:03:15.764694 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:15.764659    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"]
Sep 15 22:03:15.765249 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:15.764923    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg" podUID="bb690595-954f-4b8d-b5ee-ca7af3779ac2" containerName="istio-proxy" containerID="cri-o://031c1f90aa8155799e1c0fc7214f769acd45b43d836e7fcd3d965a0862be7ce2" gracePeriod=30
Sep 15 22:03:16.096248 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.096217    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj"]
Sep 15 22:03:16.096618 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.096605    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17f095e5-a6e0-4fff-9ccb-36ded38233a1" containerName="payload-pre-processing"
Sep 15 22:03:16.096666 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.096619    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="17f095e5-a6e0-4fff-9ccb-36ded38233a1" containerName="payload-pre-processing"
Sep 15 22:03:16.096666 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.096636    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c17a0d-49c8-4126-88a2-664e666d84d0" containerName="payload-processing"
Sep 15 22:03:16.096666 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.096642    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c17a0d-49c8-4126-88a2-664e666d84d0" containerName="payload-processing"
Sep 15 22:03:16.096666 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.096651    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42beaef4-865f-4831-8a1d-30fc885724ff" containerName="istio-proxy"
Sep 15 22:03:16.096666 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.096656    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="42beaef4-865f-4831-8a1d-30fc885724ff" containerName="istio-proxy"
Sep 15 22:03:16.096666 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.096666    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94222d30-21e4-42ea-8aee-74c616915273" containerName="revoke-keys"
Sep 15 22:03:16.096890 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.096673    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="94222d30-21e4-42ea-8aee-74c616915273" containerName="revoke-keys"
Sep 15 22:03:16.096890 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.096735    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="94222d30-21e4-42ea-8aee-74c616915273" containerName="revoke-keys"
Sep 15 22:03:16.096890 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.096742    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="42beaef4-865f-4831-8a1d-30fc885724ff" containerName="istio-proxy"
Sep 15 22:03:16.096890 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.096748    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c17a0d-49c8-4126-88a2-664e666d84d0" containerName="payload-processing"
Sep 15 22:03:16.096890 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.096770    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="42beaef4-865f-4831-8a1d-30fc885724ff" containerName="istio-proxy"
Sep 15 22:03:16.096890 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.096779    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="17f095e5-a6e0-4fff-9ccb-36ded38233a1" containerName="payload-pre-processing"
Sep 15 22:03:16.101542 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.101526    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj"
Sep 15 22:03:16.104030 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.104008    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 22:03:16.104154 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.104029    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:03:16.108010 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.107986    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj"]
Sep 15 22:03:16.218815 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.218780    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd0b9f4-11cd-4f94-862b-77279aa17f17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj\" (UID: \"6bd0b9f4-11cd-4f94-862b-77279aa17f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj"
Sep 15 22:03:16.319590 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.319556    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd0b9f4-11cd-4f94-862b-77279aa17f17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj\" (UID: \"6bd0b9f4-11cd-4f94-862b-77279aa17f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj"
Sep 15 22:03:16.320265 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.320245    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd0b9f4-11cd-4f94-862b-77279aa17f17-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj\" (UID: \"6bd0b9f4-11cd-4f94-862b-77279aa17f17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj"
Sep 15 22:03:16.414639 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.414553    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj"
Sep 15 22:03:16.687056 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:16.687030    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj"]
Sep 15 22:03:16.689744 ip-10-0-135-148 kubenswrapper[2575]: W0915 22:03:16.689715    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bd0b9f4_11cd_4f94_862b_77279aa17f17.slice/crio-fffff7ae358184322ed30916239a743366035c0f37b7d827a5070d8009ea7272 WatchSource:0}: Error finding container fffff7ae358184322ed30916239a743366035c0f37b7d827a5070d8009ea7272: Status 404 returned error can't find the container with id fffff7ae358184322ed30916239a743366035c0f37b7d827a5070d8009ea7272
Sep 15 22:03:17.307312 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:17.307277    2575 generic.go:356] "Generic (PLEG): container finished" podID="6bd0b9f4-11cd-4f94-862b-77279aa17f17" containerID="bf7a0187727b600a0b5cb8c1717915ad100ef7ff0e9c954fc5aa22b10c92dca6" exitCode=0
Sep 15 22:03:17.307662 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:17.307364    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj" event={"ID":"6bd0b9f4-11cd-4f94-862b-77279aa17f17","Type":"ContainerDied","Data":"bf7a0187727b600a0b5cb8c1717915ad100ef7ff0e9c954fc5aa22b10c92dca6"}
Sep 15 22:03:17.307662 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:17.307415    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj" event={"ID":"6bd0b9f4-11cd-4f94-862b-77279aa17f17","Type":"ContainerStarted","Data":"fffff7ae358184322ed30916239a743366035c0f37b7d827a5070d8009ea7272"}
Sep 15 22:03:18.443526 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:18.443504    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj"
Sep 15 22:03:18.539143 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:18.539109    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd0b9f4-11cd-4f94-862b-77279aa17f17-maas-api-service-ca\") pod \"6bd0b9f4-11cd-4f94-862b-77279aa17f17\" (UID: \"6bd0b9f4-11cd-4f94-862b-77279aa17f17\") "
Sep 15 22:03:18.539470 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:18.539447    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bd0b9f4-11cd-4f94-862b-77279aa17f17-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6bd0b9f4-11cd-4f94-862b-77279aa17f17" (UID: "6bd0b9f4-11cd-4f94-862b-77279aa17f17"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:03:18.640259 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:18.640228    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bd0b9f4-11cd-4f94-862b-77279aa17f17-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:19.319799 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:19.319771    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj"
Sep 15 22:03:19.319973 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:19.319776    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj" event={"ID":"6bd0b9f4-11cd-4f94-862b-77279aa17f17","Type":"ContainerDied","Data":"fffff7ae358184322ed30916239a743366035c0f37b7d827a5070d8009ea7272"}
Sep 15 22:03:19.319973 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:19.319893    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fffff7ae358184322ed30916239a743366035c0f37b7d827a5070d8009ea7272"
Sep 15 22:03:21.024071 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.024048    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 22:03:21.165746 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.165712    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-envoy\") pod \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") "
Sep 15 22:03:21.165939 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.165800    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-workload-socket\") pod \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") "
Sep 15 22:03:21.165939 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.165835    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-data\") pod \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") "
Sep 15 22:03:21.165939 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.165877    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-credential-socket\") pod \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") "
Sep 15 22:03:21.165939 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.165898    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-podinfo\") pod \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") "
Sep 15 22:03:21.165939 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.165913    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j25c\" (UniqueName: \"kubernetes.io/projected/bb690595-954f-4b8d-b5ee-ca7af3779ac2-kube-api-access-8j25c\") pod \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") "
Sep 15 22:03:21.165939 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.165941    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istiod-ca-cert\") pod \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") "
Sep 15 22:03:21.166238 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.165987    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-token\") pod \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") "
Sep 15 22:03:21.166238 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.166027    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-workload-certs\") pod \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\" (UID: \"bb690595-954f-4b8d-b5ee-ca7af3779ac2\") "
Sep 15 22:03:21.166238 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.166089    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bb690595-954f-4b8d-b5ee-ca7af3779ac2" (UID: "bb690595-954f-4b8d-b5ee-ca7af3779ac2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:03:21.166238 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.166171    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bb690595-954f-4b8d-b5ee-ca7af3779ac2" (UID: "bb690595-954f-4b8d-b5ee-ca7af3779ac2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:03:21.166238 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.166180    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-data" (OuterVolumeSpecName: "istio-data") pod "bb690595-954f-4b8d-b5ee-ca7af3779ac2" (UID: "bb690595-954f-4b8d-b5ee-ca7af3779ac2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:03:21.166513 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.166361    2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-workload-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:21.166513 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.166380    2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-data\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:21.166513 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.166394    2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-credential-socket\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:21.166513 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.166399    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bb690595-954f-4b8d-b5ee-ca7af3779ac2" (UID: "bb690595-954f-4b8d-b5ee-ca7af3779ac2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:03:21.166513 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.166438    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bb690595-954f-4b8d-b5ee-ca7af3779ac2" (UID: "bb690595-954f-4b8d-b5ee-ca7af3779ac2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:03:21.168235 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.168210    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bb690595-954f-4b8d-b5ee-ca7af3779ac2" (UID: "bb690595-954f-4b8d-b5ee-ca7af3779ac2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:03:21.168381 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.168233    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-token" (OuterVolumeSpecName: "istio-token") pod "bb690595-954f-4b8d-b5ee-ca7af3779ac2" (UID: "bb690595-954f-4b8d-b5ee-ca7af3779ac2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:03:21.168381 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.168237    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bb690595-954f-4b8d-b5ee-ca7af3779ac2" (UID: "bb690595-954f-4b8d-b5ee-ca7af3779ac2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 22:03:21.168480 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.168387    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb690595-954f-4b8d-b5ee-ca7af3779ac2-kube-api-access-8j25c" (OuterVolumeSpecName: "kube-api-access-8j25c") pod "bb690595-954f-4b8d-b5ee-ca7af3779ac2" (UID: "bb690595-954f-4b8d-b5ee-ca7af3779ac2"). InnerVolumeSpecName "kube-api-access-8j25c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:03:21.267732 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.267696    2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istiod-ca-cert\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:21.267732 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.267724    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-token\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:21.267732 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.267732    2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-workload-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:21.267732 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.267740    2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-envoy\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:21.268018 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.267747    2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb690595-954f-4b8d-b5ee-ca7af3779ac2-istio-podinfo\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:21.268018 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.267769    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j25c\" (UniqueName: \"kubernetes.io/projected/bb690595-954f-4b8d-b5ee-ca7af3779ac2-kube-api-access-8j25c\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:21.331517 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.331474    2575 generic.go:356] "Generic (PLEG): container finished" podID="bb690595-954f-4b8d-b5ee-ca7af3779ac2" containerID="031c1f90aa8155799e1c0fc7214f769acd45b43d836e7fcd3d965a0862be7ce2" exitCode=0
Sep 15 22:03:21.331703 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.331529    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"
Sep 15 22:03:21.331703 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.331568    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg" event={"ID":"bb690595-954f-4b8d-b5ee-ca7af3779ac2","Type":"ContainerDied","Data":"031c1f90aa8155799e1c0fc7214f769acd45b43d836e7fcd3d965a0862be7ce2"}
Sep 15 22:03:21.331703 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.331624    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg" event={"ID":"bb690595-954f-4b8d-b5ee-ca7af3779ac2","Type":"ContainerDied","Data":"4119866ac7fa8c2c2727b5c386bac8ac64a223252480629478202f419c04053b"}
Sep 15 22:03:21.331703 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.331647    2575 scope.go:117] "RemoveContainer" containerID="031c1f90aa8155799e1c0fc7214f769acd45b43d836e7fcd3d965a0862be7ce2"
Sep 15 22:03:21.341631 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.341614    2575 scope.go:117] "RemoveContainer" containerID="b2f92cd2ae46616efcb701cd226359a02e0bb50874145b39a596e50f217f37f9"
Sep 15 22:03:21.350262 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.350241    2575 scope.go:117] "RemoveContainer" containerID="031c1f90aa8155799e1c0fc7214f769acd45b43d836e7fcd3d965a0862be7ce2"
Sep 15 22:03:21.350907 ip-10-0-135-148 kubenswrapper[2575]: E0915 22:03:21.350578    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031c1f90aa8155799e1c0fc7214f769acd45b43d836e7fcd3d965a0862be7ce2\": container with ID starting with 031c1f90aa8155799e1c0fc7214f769acd45b43d836e7fcd3d965a0862be7ce2 not found: ID does not exist" containerID="031c1f90aa8155799e1c0fc7214f769acd45b43d836e7fcd3d965a0862be7ce2"
Sep 15 22:03:21.350907 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.350617    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031c1f90aa8155799e1c0fc7214f769acd45b43d836e7fcd3d965a0862be7ce2"} err="failed to get container status \"031c1f90aa8155799e1c0fc7214f769acd45b43d836e7fcd3d965a0862be7ce2\": rpc error: code = NotFound desc = could not find container \"031c1f90aa8155799e1c0fc7214f769acd45b43d836e7fcd3d965a0862be7ce2\": container with ID starting with 031c1f90aa8155799e1c0fc7214f769acd45b43d836e7fcd3d965a0862be7ce2 not found: ID does not exist"
Sep 15 22:03:21.350907 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.350668    2575 scope.go:117] "RemoveContainer" containerID="b2f92cd2ae46616efcb701cd226359a02e0bb50874145b39a596e50f217f37f9"
Sep 15 22:03:21.351100 ip-10-0-135-148 kubenswrapper[2575]: E0915 22:03:21.350998    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f92cd2ae46616efcb701cd226359a02e0bb50874145b39a596e50f217f37f9\": container with ID starting with b2f92cd2ae46616efcb701cd226359a02e0bb50874145b39a596e50f217f37f9 not found: ID does not exist" containerID="b2f92cd2ae46616efcb701cd226359a02e0bb50874145b39a596e50f217f37f9"
Sep 15 22:03:21.351100 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.351027    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f92cd2ae46616efcb701cd226359a02e0bb50874145b39a596e50f217f37f9"} err="failed to get container status \"b2f92cd2ae46616efcb701cd226359a02e0bb50874145b39a596e50f217f37f9\": rpc error: code = NotFound desc = could not find container \"b2f92cd2ae46616efcb701cd226359a02e0bb50874145b39a596e50f217f37f9\": container with ID starting with b2f92cd2ae46616efcb701cd226359a02e0bb50874145b39a596e50f217f37f9 not found: ID does not exist"
Sep 15 22:03:21.353060 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.353040    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"]
Sep 15 22:03:21.359128 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:21.359106    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1297a3-openshift-default-5d86fcf6d6-gh4lg"]
Sep 15 22:03:22.747818 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:22.747788    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb690595-954f-4b8d-b5ee-ca7af3779ac2" path="/var/lib/kubelet/pods/bb690595-954f-4b8d-b5ee-ca7af3779ac2/volumes"
Sep 15 22:03:26.508463 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:26.508386    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce3ab0-29825160-kjx92"]
Sep 15 22:03:26.511940 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:26.511915    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ce3ab0-29825160-kjx92"]
Sep 15 22:03:26.729783 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:26.729741    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj"]
Sep 15 22:03:26.733482 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:26.733453    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ce3ab0-crjvj"]
Sep 15 22:03:26.748276 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:26.748251    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bd0b9f4-11cd-4f94-862b-77279aa17f17" path="/var/lib/kubelet/pods/6bd0b9f4-11cd-4f94-862b-77279aa17f17/volumes"
Sep 15 22:03:26.748526 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:26.748514    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f84691e5-184c-4031-a84c-481afcc111b3" path="/var/lib/kubelet/pods/f84691e5-184c-4031-a84c-481afcc111b3/volumes"
Sep 15 22:03:29.457599 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:29.457564    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:03:33.718683 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.718648    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689"]
Sep 15 22:03:33.719080 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.719052    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bd0b9f4-11cd-4f94-862b-77279aa17f17" containerName="revoke-keys"
Sep 15 22:03:33.719080 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.719064    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bd0b9f4-11cd-4f94-862b-77279aa17f17" containerName="revoke-keys"
Sep 15 22:03:33.719080 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.719078    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42beaef4-865f-4831-8a1d-30fc885724ff" containerName="istio-proxy"
Sep 15 22:03:33.719188 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.719083    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="42beaef4-865f-4831-8a1d-30fc885724ff" containerName="istio-proxy"
Sep 15 22:03:33.719188 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.719114    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb690595-954f-4b8d-b5ee-ca7af3779ac2" containerName="istio-proxy"
Sep 15 22:03:33.719188 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.719122    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb690595-954f-4b8d-b5ee-ca7af3779ac2" containerName="istio-proxy"
Sep 15 22:03:33.719188 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.719133    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb690595-954f-4b8d-b5ee-ca7af3779ac2" containerName="istio-proxy"
Sep 15 22:03:33.719188 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.719141    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb690595-954f-4b8d-b5ee-ca7af3779ac2" containerName="istio-proxy"
Sep 15 22:03:33.719342 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.719223    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb690595-954f-4b8d-b5ee-ca7af3779ac2" containerName="istio-proxy"
Sep 15 22:03:33.719342 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.719234    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb690595-954f-4b8d-b5ee-ca7af3779ac2" containerName="istio-proxy"
Sep 15 22:03:33.719342 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.719243    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bd0b9f4-11cd-4f94-862b-77279aa17f17" containerName="revoke-keys"
Sep 15 22:03:33.723851 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.723835    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689"
Sep 15 22:03:33.726373 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.726355    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-p7cdh\""
Sep 15 22:03:33.726505 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.726357    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 22:03:33.730213 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.730145    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689"]
Sep 15 22:03:33.890082 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.890043    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2cb017-a944-4c02-95aa-af0785728724-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689\" (UID: \"2b2cb017-a944-4c02-95aa-af0785728724\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689"
Sep 15 22:03:33.991417 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.991331    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2cb017-a944-4c02-95aa-af0785728724-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689\" (UID: \"2b2cb017-a944-4c02-95aa-af0785728724\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689"
Sep 15 22:03:33.991971 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:33.991953    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2cb017-a944-4c02-95aa-af0785728724-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689\" (UID: \"2b2cb017-a944-4c02-95aa-af0785728724\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689"
Sep 15 22:03:34.035443 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:34.035413    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689"
Sep 15 22:03:34.306894 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:34.306867    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689"]
Sep 15 22:03:34.308798 ip-10-0-135-148 kubenswrapper[2575]: W0915 22:03:34.308748    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2cb017_a944_4c02_95aa_af0785728724.slice/crio-cd009456548b8a0f305e9cd0238f3f5999606a7803eec68b699110f105861972 WatchSource:0}: Error finding container cd009456548b8a0f305e9cd0238f3f5999606a7803eec68b699110f105861972: Status 404 returned error can't find the container with id cd009456548b8a0f305e9cd0238f3f5999606a7803eec68b699110f105861972
Sep 15 22:03:34.403170 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:34.403136    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689" event={"ID":"2b2cb017-a944-4c02-95aa-af0785728724","Type":"ContainerStarted","Data":"cd009456548b8a0f305e9cd0238f3f5999606a7803eec68b699110f105861972"}
Sep 15 22:03:35.410096 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:35.410063    2575 generic.go:356] "Generic (PLEG): container finished" podID="2b2cb017-a944-4c02-95aa-af0785728724" containerID="58112d593d2e07607a915deb851dc1f0f2f36d0e253fd0a92ce886ca19f8fda1" exitCode=0
Sep 15 22:03:35.410526 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:35.410150    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689" event={"ID":"2b2cb017-a944-4c02-95aa-af0785728724","Type":"ContainerDied","Data":"58112d593d2e07607a915deb851dc1f0f2f36d0e253fd0a92ce886ca19f8fda1"}
Sep 15 22:03:36.553181 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:36.553155    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689"
Sep 15 22:03:36.615325 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:36.615292    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2cb017-a944-4c02-95aa-af0785728724-maas-api-service-ca\") pod \"2b2cb017-a944-4c02-95aa-af0785728724\" (UID: \"2b2cb017-a944-4c02-95aa-af0785728724\") "
Sep 15 22:03:36.615709 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:36.615687    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b2cb017-a944-4c02-95aa-af0785728724-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b2cb017-a944-4c02-95aa-af0785728724" (UID: "2b2cb017-a944-4c02-95aa-af0785728724"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:03:36.716726 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:36.716639    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b2cb017-a944-4c02-95aa-af0785728724-maas-api-service-ca\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:37.423025 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:37.422991    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689" event={"ID":"2b2cb017-a944-4c02-95aa-af0785728724","Type":"ContainerDied","Data":"cd009456548b8a0f305e9cd0238f3f5999606a7803eec68b699110f105861972"}
Sep 15 22:03:37.423025 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:37.423017    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689"
Sep 15 22:03:37.423238 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:37.423039    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd009456548b8a0f305e9cd0238f3f5999606a7803eec68b699110f105861972"
Sep 15 22:03:44.376339 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:44.376304    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e7a648-29825160-cv4p8"]
Sep 15 22:03:44.379328 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:44.379304    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e7a648-29825160-cv4p8"]
Sep 15 22:03:44.412720 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:44.412687    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"]
Sep 15 22:03:44.413029 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:44.412980    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k" podUID="f433221e-a8c8-407c-9eaa-549526cdb6f0" containerName="payload-pre-processing" containerID="cri-o://561b1d977224b1ac54b2d251ce176196b594879a339d9b3d2302339359f9acab" gracePeriod=30
Sep 15 22:03:44.620332 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:44.620300    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689"]
Sep 15 22:03:44.623626 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:44.623604    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e7a648-4d689"]
Sep 15 22:03:44.657255 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:44.657237    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"
Sep 15 22:03:44.683988 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:44.683966    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5zcs\" (UniqueName: \"kubernetes.io/projected/f433221e-a8c8-407c-9eaa-549526cdb6f0-kube-api-access-b5zcs\") pod \"f433221e-a8c8-407c-9eaa-549526cdb6f0\" (UID: \"f433221e-a8c8-407c-9eaa-549526cdb6f0\") "
Sep 15 22:03:44.684129 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:44.684016    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f433221e-a8c8-407c-9eaa-549526cdb6f0-plugins-config-volume\") pod \"f433221e-a8c8-407c-9eaa-549526cdb6f0\" (UID: \"f433221e-a8c8-407c-9eaa-549526cdb6f0\") "
Sep 15 22:03:44.684457 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:44.684429    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f433221e-a8c8-407c-9eaa-549526cdb6f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f433221e-a8c8-407c-9eaa-549526cdb6f0" (UID: "f433221e-a8c8-407c-9eaa-549526cdb6f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 22:03:44.686469 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:44.686430    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f433221e-a8c8-407c-9eaa-549526cdb6f0-kube-api-access-b5zcs" (OuterVolumeSpecName: "kube-api-access-b5zcs") pod "f433221e-a8c8-407c-9eaa-549526cdb6f0" (UID: "f433221e-a8c8-407c-9eaa-549526cdb6f0"). InnerVolumeSpecName "kube-api-access-b5zcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:03:44.746850 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:44.746819    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b2cb017-a944-4c02-95aa-af0785728724" path="/var/lib/kubelet/pods/2b2cb017-a944-4c02-95aa-af0785728724/volumes"
Sep 15 22:03:44.747129 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:44.747116    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9050b59e-6373-4ae9-8c2a-5cfe8d7be911" path="/var/lib/kubelet/pods/9050b59e-6373-4ae9-8c2a-5cfe8d7be911/volumes"
Sep 15 22:03:44.785339 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:44.785311    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5zcs\" (UniqueName: \"kubernetes.io/projected/f433221e-a8c8-407c-9eaa-549526cdb6f0-kube-api-access-b5zcs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:44.785339 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:44.785336    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f433221e-a8c8-407c-9eaa-549526cdb6f0-plugins-config-volume\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:03:45.463899 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:45.463864    2575 generic.go:356] "Generic (PLEG): container finished" podID="f433221e-a8c8-407c-9eaa-549526cdb6f0" containerID="561b1d977224b1ac54b2d251ce176196b594879a339d9b3d2302339359f9acab" exitCode=0
Sep 15 22:03:45.464298 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:45.463907    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"
Sep 15 22:03:45.464298 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:45.463951    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k" event={"ID":"f433221e-a8c8-407c-9eaa-549526cdb6f0","Type":"ContainerDied","Data":"561b1d977224b1ac54b2d251ce176196b594879a339d9b3d2302339359f9acab"}
Sep 15 22:03:45.464298 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:45.464004    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k" event={"ID":"f433221e-a8c8-407c-9eaa-549526cdb6f0","Type":"ContainerDied","Data":"5bd65bc7fb25d152ed99df20401078eadc25bf0cab77b57112a6089c649ca9f8"}
Sep 15 22:03:45.464298 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:45.464027    2575 scope.go:117] "RemoveContainer" containerID="561b1d977224b1ac54b2d251ce176196b594879a339d9b3d2302339359f9acab"
Sep 15 22:03:45.472783 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:45.472766    2575 scope.go:117] "RemoveContainer" containerID="561b1d977224b1ac54b2d251ce176196b594879a339d9b3d2302339359f9acab"
Sep 15 22:03:45.473017 ip-10-0-135-148 kubenswrapper[2575]: E0915 22:03:45.472998    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561b1d977224b1ac54b2d251ce176196b594879a339d9b3d2302339359f9acab\": container with ID starting with 561b1d977224b1ac54b2d251ce176196b594879a339d9b3d2302339359f9acab not found: ID does not exist" containerID="561b1d977224b1ac54b2d251ce176196b594879a339d9b3d2302339359f9acab"
Sep 15 22:03:45.473078 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:45.473026    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561b1d977224b1ac54b2d251ce176196b594879a339d9b3d2302339359f9acab"} err="failed to get container status \"561b1d977224b1ac54b2d251ce176196b594879a339d9b3d2302339359f9acab\": rpc error: code = NotFound desc = could not find container \"561b1d977224b1ac54b2d251ce176196b594879a339d9b3d2302339359f9acab\": container with ID starting with 561b1d977224b1ac54b2d251ce176196b594879a339d9b3d2302339359f9acab not found: ID does not exist"
Sep 15 22:03:45.480105 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:45.480084    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"]
Sep 15 22:03:45.485647 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:45.485627    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e7a648-6579dfb5db-tg87k"]
Sep 15 22:03:46.747953 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:46.747924    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f433221e-a8c8-407c-9eaa-549526cdb6f0" path="/var/lib/kubelet/pods/f433221e-a8c8-407c-9eaa-549526cdb6f0/volumes"
Sep 15 22:03:47.440034 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:47.439998    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:03:55.430653 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:03:55.430617    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:05:16.341145 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:05:16.341107    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:05:40.539422 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:05:40.539388    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:05:48.542690 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:05:48.542650    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:05:55.938898 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:05:55.938858    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:06:02.940991 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:06:02.940955    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:06:04.099952 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:06:04.099920    2575 scope.go:117] "RemoveContainer" containerID="e25ad19a56e4d6df5164017fe901231fbf1713453b60df74a22aa1f6cac5e8e7"
Sep 15 22:06:04.110643 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:06:04.110614    2575 scope.go:117] "RemoveContainer" containerID="315a417a00c816f30f2c58aef2cd1060467942e1a9de491fbbb63ce4fccff474"
Sep 15 22:06:04.118681 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:06:04.118659    2575 scope.go:117] "RemoveContainer" containerID="0198efc69ee803da6b21dd9c9f1088a397c5c854f3fd34df866abbe46fc052c2"
Sep 15 22:06:17.642653 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:06:17.642620    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:06:25.627709 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:06:25.627632    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:07:52.840189 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:07:52.840152    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:07:59.565292 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:07:59.565255    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:08:04.151779 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:04.151730    2575 scope.go:117] "RemoveContainer" containerID="ade3c345ca7aa04c8fa805c9c13600cfaf249a1db0db850c63a52106fe342ccb"
Sep 15 22:08:04.160441 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:04.160420    2575 scope.go:117] "RemoveContainer" containerID="844b44f41a738654367d36a3f90adbb0f7e2ad8f248cf565e282c12577a57812"
Sep 15 22:08:08.940517 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:08.940480    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:08:13.249621 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.249593    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"]
Sep 15 22:08:13.250043 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.250023    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b2cb017-a944-4c02-95aa-af0785728724" containerName="revoke-keys"
Sep 15 22:08:13.250043 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.250036    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b2cb017-a944-4c02-95aa-af0785728724" containerName="revoke-keys"
Sep 15 22:08:13.250122 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.250064    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f433221e-a8c8-407c-9eaa-549526cdb6f0" containerName="payload-pre-processing"
Sep 15 22:08:13.250122 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.250069    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f433221e-a8c8-407c-9eaa-549526cdb6f0" containerName="payload-pre-processing"
Sep 15 22:08:13.250190 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.250128    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f433221e-a8c8-407c-9eaa-549526cdb6f0" containerName="payload-pre-processing"
Sep 15 22:08:13.250190 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.250139    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b2cb017-a944-4c02-95aa-af0785728724" containerName="revoke-keys"
Sep 15 22:08:13.253224 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.253206    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.255387 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.255365    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-fbb927c8-kserve-self-signed-certs\""
Sep 15 22:08:13.265824 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.265791    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"]
Sep 15 22:08:13.348611 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.348571    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-tmp-dir\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.348818 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.348636    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb442\" (UniqueName: \"kubernetes.io/projected/64b985df-1a3d-4471-a90d-972147469002-kube-api-access-fb442\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.348818 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.348692    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-dshm\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.348818 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.348713    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b985df-1a3d-4471-a90d-972147469002-tls-certs\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.348928 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.348835    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-model-cache\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.348928 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.348872    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-home\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.449463 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.449433    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-home\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.449650 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.449471    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-tmp-dir\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.449650 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.449519    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb442\" (UniqueName: \"kubernetes.io/projected/64b985df-1a3d-4471-a90d-972147469002-kube-api-access-fb442\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.449650 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.449562    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-dshm\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.449650 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.449597    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b985df-1a3d-4471-a90d-972147469002-tls-certs\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.449650 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.449638    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-model-cache\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.449948 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.449930    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-home\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.450007 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.449963    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-tmp-dir\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.450007 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.449991    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-model-cache\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.452004 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.451981    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-dshm\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.452251 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.452232    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b985df-1a3d-4471-a90d-972147469002-tls-certs\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.458266 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.458239    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb442\" (UniqueName: \"kubernetes.io/projected/64b985df-1a3d-4471-a90d-972147469002-kube-api-access-fb442\") pod \"e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") " pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.567899 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.567811    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:13.842151 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:13.842126    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"]
Sep 15 22:08:13.844714 ip-10-0-135-148 kubenswrapper[2575]: W0915 22:08:13.844684    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64b985df_1a3d_4471_a90d_972147469002.slice/crio-730add209255888e09f9ce9e4df0189d61b1d76951d93ff0e149338badd19e58 WatchSource:0}: Error finding container 730add209255888e09f9ce9e4df0189d61b1d76951d93ff0e149338badd19e58: Status 404 returned error can't find the container with id 730add209255888e09f9ce9e4df0189d61b1d76951d93ff0e149338badd19e58
Sep 15 22:08:14.844903 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:14.844864    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd" event={"ID":"64b985df-1a3d-4471-a90d-972147469002","Type":"ContainerStarted","Data":"01e373ef1444fe387ef143200d8cbc2026b2de4a6e2be7f815e308cdb1d89dbf"}
Sep 15 22:08:14.845309 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:14.844912    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd" event={"ID":"64b985df-1a3d-4471-a90d-972147469002","Type":"ContainerStarted","Data":"730add209255888e09f9ce9e4df0189d61b1d76951d93ff0e149338badd19e58"}
Sep 15 22:08:14.870065 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:14.870012    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd" podStartSLOduration=1.8699960930000001 podStartE2EDuration="1.869996093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:08:14.867090003 +0000 UTC m=+2532.679098447" watchObservedRunningTime="2026-09-15 22:08:14.869996093 +0000 UTC m=+2532.682004537"
Sep 15 22:08:16.673078 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:16.673044    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:08:23.568558 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:23.568517    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:23.568558 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:23.568563    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:23.581233 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:23.581206    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:23.908176 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:23.908141    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:08:56.959526 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:08:56.959483    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:09:04.183092 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:04.183062    2575 scope.go:117] "RemoveContainer" containerID="8f77ecfe57ac562709f06b5f47577861a7d7e7d1ce7abf5df3da65711f7386ac"
Sep 15 22:09:09.330375 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:09.330331    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"]
Sep 15 22:09:11.548545 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:11.548514    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:09:20.653583 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:20.653545    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:09:24.350146 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.350053    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd" podUID="64b985df-1a3d-4471-a90d-972147469002" containerName="main" containerID="cri-o://01e373ef1444fe387ef143200d8cbc2026b2de4a6e2be7f815e308cdb1d89dbf" gracePeriod=45
Sep 15 22:09:24.602689 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.602634    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:09:24.716235 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.716204    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b985df-1a3d-4471-a90d-972147469002-tls-certs\") pod \"64b985df-1a3d-4471-a90d-972147469002\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") "
Sep 15 22:09:24.716442 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.716265    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-home\") pod \"64b985df-1a3d-4471-a90d-972147469002\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") "
Sep 15 22:09:24.716442 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.716317    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fb442\" (UniqueName: \"kubernetes.io/projected/64b985df-1a3d-4471-a90d-972147469002-kube-api-access-fb442\") pod \"64b985df-1a3d-4471-a90d-972147469002\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") "
Sep 15 22:09:24.716442 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.716354    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-tmp-dir\") pod \"64b985df-1a3d-4471-a90d-972147469002\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") "
Sep 15 22:09:24.716442 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.716401    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-dshm\") pod \"64b985df-1a3d-4471-a90d-972147469002\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") "
Sep 15 22:09:24.716669 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.716448    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-model-cache\") pod \"64b985df-1a3d-4471-a90d-972147469002\" (UID: \"64b985df-1a3d-4471-a90d-972147469002\") "
Sep 15 22:09:24.716669 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.716558    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-home" (OuterVolumeSpecName: "home") pod "64b985df-1a3d-4471-a90d-972147469002" (UID: "64b985df-1a3d-4471-a90d-972147469002"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:09:24.716799 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.716715    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64b985df-1a3d-4471-a90d-972147469002" (UID: "64b985df-1a3d-4471-a90d-972147469002"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:09:24.716799 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.716722    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-model-cache" (OuterVolumeSpecName: "model-cache") pod "64b985df-1a3d-4471-a90d-972147469002" (UID: "64b985df-1a3d-4471-a90d-972147469002"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:09:24.716881 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.716871    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-home\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:09:24.716916 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.716888    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-tmp-dir\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:09:24.716916 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.716902    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-model-cache\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:09:24.718704 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.718679    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64b985df-1a3d-4471-a90d-972147469002-kube-api-access-fb442" (OuterVolumeSpecName: "kube-api-access-fb442") pod "64b985df-1a3d-4471-a90d-972147469002" (UID: "64b985df-1a3d-4471-a90d-972147469002"). InnerVolumeSpecName "kube-api-access-fb442". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 22:09:24.718825 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.718714    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64b985df-1a3d-4471-a90d-972147469002-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64b985df-1a3d-4471-a90d-972147469002" (UID: "64b985df-1a3d-4471-a90d-972147469002"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 22:09:24.718825 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.718714    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-dshm" (OuterVolumeSpecName: "dshm") pod "64b985df-1a3d-4471-a90d-972147469002" (UID: "64b985df-1a3d-4471-a90d-972147469002"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 22:09:24.817396 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.817364    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb442\" (UniqueName: \"kubernetes.io/projected/64b985df-1a3d-4471-a90d-972147469002-kube-api-access-fb442\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:09:24.817396 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.817390    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64b985df-1a3d-4471-a90d-972147469002-dshm\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:09:24.817396 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:24.817399    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64b985df-1a3d-4471-a90d-972147469002-tls-certs\") on node \"ip-10-0-135-148.ec2.internal\" DevicePath \"\""
Sep 15 22:09:25.193054 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:25.193018    2575 generic.go:356] "Generic (PLEG): container finished" podID="64b985df-1a3d-4471-a90d-972147469002" containerID="01e373ef1444fe387ef143200d8cbc2026b2de4a6e2be7f815e308cdb1d89dbf" exitCode=0
Sep 15 22:09:25.193243 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:25.193060    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"
Sep 15 22:09:25.193243 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:25.193105    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd" event={"ID":"64b985df-1a3d-4471-a90d-972147469002","Type":"ContainerDied","Data":"01e373ef1444fe387ef143200d8cbc2026b2de4a6e2be7f815e308cdb1d89dbf"}
Sep 15 22:09:25.193243 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:25.193152    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd" event={"ID":"64b985df-1a3d-4471-a90d-972147469002","Type":"ContainerDied","Data":"730add209255888e09f9ce9e4df0189d61b1d76951d93ff0e149338badd19e58"}
Sep 15 22:09:25.193243 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:25.193174    2575 scope.go:117] "RemoveContainer" containerID="01e373ef1444fe387ef143200d8cbc2026b2de4a6e2be7f815e308cdb1d89dbf"
Sep 15 22:09:25.208247 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:25.208228    2575 scope.go:117] "RemoveContainer" containerID="01e373ef1444fe387ef143200d8cbc2026b2de4a6e2be7f815e308cdb1d89dbf"
Sep 15 22:09:25.208529 ip-10-0-135-148 kubenswrapper[2575]: E0915 22:09:25.208508    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e373ef1444fe387ef143200d8cbc2026b2de4a6e2be7f815e308cdb1d89dbf\": container with ID starting with 01e373ef1444fe387ef143200d8cbc2026b2de4a6e2be7f815e308cdb1d89dbf not found: ID does not exist" containerID="01e373ef1444fe387ef143200d8cbc2026b2de4a6e2be7f815e308cdb1d89dbf"
Sep 15 22:09:25.208604 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:25.208538    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e373ef1444fe387ef143200d8cbc2026b2de4a6e2be7f815e308cdb1d89dbf"} err="failed to get container status \"01e373ef1444fe387ef143200d8cbc2026b2de4a6e2be7f815e308cdb1d89dbf\": rpc error: code = NotFound desc = could not find container \"01e373ef1444fe387ef143200d8cbc2026b2de4a6e2be7f815e308cdb1d89dbf\": container with ID starting with 01e373ef1444fe387ef143200d8cbc2026b2de4a6e2be7f815e308cdb1d89dbf not found: ID does not exist"
Sep 15 22:09:25.222234 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:25.222207    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"]
Sep 15 22:09:25.226148 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:25.226125    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-fbb927c8-kserve-78578588ff-n8cnd"]
Sep 15 22:09:26.747817 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:26.747785    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64b985df-1a3d-4471-a90d-972147469002" path="/var/lib/kubelet/pods/64b985df-1a3d-4471-a90d-972147469002/volumes"
Sep 15 22:09:27.952721 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:27.952680    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:09:34.242299 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:34.242266    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:09:42.257148 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:42.257113    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:09:49.440673 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:49.440631    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:09:55.172671 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:09:55.172634    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:10:02.042224 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:02.042191    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-shtqn"]
Sep 15 22:10:04.202577 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:04.202544    2575 scope.go:117] "RemoveContainer" containerID="58112d593d2e07607a915deb851dc1f0f2f36d0e253fd0a92ce886ca19f8fda1"
Sep 15 22:10:04.211024 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:04.210996    2575 scope.go:117] "RemoveContainer" containerID="bf7a0187727b600a0b5cb8c1717915ad100ef7ff0e9c954fc5aa22b10c92dca6"
Sep 15 22:10:11.939603 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:11.939572    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77fddcc9c6-qzxzg_8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855/manager/0.log"
Sep 15 22:10:11.946249 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:11.946231    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77fddcc9c6-qzxzg_8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855/copy-manifests/0.log"
Sep 15 22:10:12.557431 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:12.557402    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29825145-mc74c_858c5f19-5948-41b1-8607-8a07a6c1ebed/cleanup/0.log"
Sep 15 22:10:14.036528 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:14.036500    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-mgcqw_1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8/kuadrant-console-plugin/0.log"
Sep 15 22:10:14.324793 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:14.324702    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-shtqn_0841737a-b3ce-4940-8e0a-0e48c5e3d882/limitador/0.log"
Sep 15 22:10:15.095560 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:15.095531    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d78dff5cc-c254b_993a3b58-915c-48cf-860e-0695b63089b4/kube-auth-proxy/0.log"
Sep 15 22:10:15.473956 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:15.473884    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xx5vp_2e58dd44-0283-4acd-a0c1-3984a3451a52/payload-pre-processing/0.log"
Sep 15 22:10:16.099028 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:16.098999    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-gj86q_06cebf8b-8c61-4f55-8a83-4b04c172ce40/main/0.log"
Sep 15 22:10:16.359697 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:16.359633    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-pgrhr_a353341f-6fe2-47ed-852e-4e2cb39de18b/main/0.log"
Sep 15 22:10:16.866413 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:16.866383    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-9flvv_98508da6-aab2-4506-92e2-7a9f7e3467a5/main/0.log"
Sep 15 22:10:22.285355 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.285324    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lgf4x/must-gather-r4tqx"]
Sep 15 22:10:22.285770 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.285740    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64b985df-1a3d-4471-a90d-972147469002" containerName="main"
Sep 15 22:10:22.285822 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.285772    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="64b985df-1a3d-4471-a90d-972147469002" containerName="main"
Sep 15 22:10:22.285858 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.285843    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="64b985df-1a3d-4471-a90d-972147469002" containerName="main"
Sep 15 22:10:22.288997 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.288977    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lgf4x/must-gather-r4tqx"
Sep 15 22:10:22.291700 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.291676    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lgf4x\"/\"default-dockercfg-5glwm\""
Sep 15 22:10:22.291829 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.291714    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lgf4x\"/\"openshift-service-ca.crt\""
Sep 15 22:10:22.292400 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.292385    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lgf4x\"/\"kube-root-ca.crt\""
Sep 15 22:10:22.304585 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.304558    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lgf4x/must-gather-r4tqx"]
Sep 15 22:10:22.350058 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.350027    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlbbd\" (UniqueName: \"kubernetes.io/projected/9667e8bc-d1e1-41fc-9ffa-5f87cbecfa9f-kube-api-access-tlbbd\") pod \"must-gather-r4tqx\" (UID: \"9667e8bc-d1e1-41fc-9ffa-5f87cbecfa9f\") " pod="openshift-must-gather-lgf4x/must-gather-r4tqx"
Sep 15 22:10:22.350246 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.350091    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9667e8bc-d1e1-41fc-9ffa-5f87cbecfa9f-must-gather-output\") pod \"must-gather-r4tqx\" (UID: \"9667e8bc-d1e1-41fc-9ffa-5f87cbecfa9f\") " pod="openshift-must-gather-lgf4x/must-gather-r4tqx"
Sep 15 22:10:22.450635 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.450596    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlbbd\" (UniqueName: \"kubernetes.io/projected/9667e8bc-d1e1-41fc-9ffa-5f87cbecfa9f-kube-api-access-tlbbd\") pod \"must-gather-r4tqx\" (UID: \"9667e8bc-d1e1-41fc-9ffa-5f87cbecfa9f\") " pod="openshift-must-gather-lgf4x/must-gather-r4tqx"
Sep 15 22:10:22.450841 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.450668    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9667e8bc-d1e1-41fc-9ffa-5f87cbecfa9f-must-gather-output\") pod \"must-gather-r4tqx\" (UID: \"9667e8bc-d1e1-41fc-9ffa-5f87cbecfa9f\") " pod="openshift-must-gather-lgf4x/must-gather-r4tqx"
Sep 15 22:10:22.451029 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.451012    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9667e8bc-d1e1-41fc-9ffa-5f87cbecfa9f-must-gather-output\") pod \"must-gather-r4tqx\" (UID: \"9667e8bc-d1e1-41fc-9ffa-5f87cbecfa9f\") " pod="openshift-must-gather-lgf4x/must-gather-r4tqx"
Sep 15 22:10:22.460509 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.460482    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlbbd\" (UniqueName: \"kubernetes.io/projected/9667e8bc-d1e1-41fc-9ffa-5f87cbecfa9f-kube-api-access-tlbbd\") pod \"must-gather-r4tqx\" (UID: \"9667e8bc-d1e1-41fc-9ffa-5f87cbecfa9f\") " pod="openshift-must-gather-lgf4x/must-gather-r4tqx"
Sep 15 22:10:22.598981 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.598955    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lgf4x/must-gather-r4tqx"
Sep 15 22:10:22.874554 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.874528    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lgf4x/must-gather-r4tqx"]
Sep 15 22:10:22.877365 ip-10-0-135-148 kubenswrapper[2575]: W0915 22:10:22.877329    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9667e8bc_d1e1_41fc_9ffa_5f87cbecfa9f.slice/crio-7707e529ba1d733ed096080426b26c856c639482cd2a1be9f2f85918cfae0a0f WatchSource:0}: Error finding container 7707e529ba1d733ed096080426b26c856c639482cd2a1be9f2f85918cfae0a0f: Status 404 returned error can't find the container with id 7707e529ba1d733ed096080426b26c856c639482cd2a1be9f2f85918cfae0a0f
Sep 15 22:10:22.879281 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:22.879261    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 22:10:23.482207 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:23.482173    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lgf4x/must-gather-r4tqx" event={"ID":"9667e8bc-d1e1-41fc-9ffa-5f87cbecfa9f","Type":"ContainerStarted","Data":"7707e529ba1d733ed096080426b26c856c639482cd2a1be9f2f85918cfae0a0f"}
Sep 15 22:10:24.490246 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:24.490202    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lgf4x/must-gather-r4tqx" event={"ID":"9667e8bc-d1e1-41fc-9ffa-5f87cbecfa9f","Type":"ContainerStarted","Data":"c93bac1aae361cf16a38f11563e6e38d19868a947141ec925c0231ba6a13cc2c"}
Sep 15 22:10:24.490811 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:24.490259    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lgf4x/must-gather-r4tqx" event={"ID":"9667e8bc-d1e1-41fc-9ffa-5f87cbecfa9f","Type":"ContainerStarted","Data":"12a92134543d58af4a7fd1f0e1c9c210905eb2603256b634e1a61f5e2057e96e"}
Sep 15 22:10:24.513467 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:24.513417    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lgf4x/must-gather-r4tqx" podStartSLOduration=1.772067313 podStartE2EDuration="2.513399228s" totalImagesPullingTime="741.331915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 22:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:10:24.511581477 +0000 UTC m=+2662.323589926" watchObservedRunningTime="2026-09-15 22:10:24.513399228 +0000 UTC m=+2662.325407672"
Sep 15 22:10:25.360680 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:25.360653    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v6j25_21784d6c-c661-42eb-8b09-41376dacc1a3/global-pull-secret-syncer/0.log"
Sep 15 22:10:25.541174 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:25.541145    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zvtxn_722562f8-7296-47bd-8fec-1d6791c5afd0/konnectivity-agent/0.log"
Sep 15 22:10:25.624883 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:25.624813    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-148.ec2.internal_ea4e7f0d7b04a6dd59e41a5028a23abd/haproxy/0.log"
Sep 15 22:10:30.159788 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:30.159739    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-mgcqw_1a3fa7b2-cf4b-4bf2-bc45-3c157683a3a8/kuadrant-console-plugin/0.log"
Sep 15 22:10:30.255368 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:30.255334    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-shtqn_0841737a-b3ce-4940-8e0a-0e48c5e3d882/limitador/0.log"
Sep 15 22:10:31.832312 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:31.832289    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5873c6d7-792c-41ac-b353-8cccb5f17a9b/alertmanager/0.log"
Sep 15 22:10:31.863142 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:31.863106    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5873c6d7-792c-41ac-b353-8cccb5f17a9b/config-reloader/0.log"
Sep 15 22:10:31.891992 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:31.891962    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5873c6d7-792c-41ac-b353-8cccb5f17a9b/kube-rbac-proxy-web/0.log"
Sep 15 22:10:31.921162 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:31.921139    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5873c6d7-792c-41ac-b353-8cccb5f17a9b/kube-rbac-proxy/0.log"
Sep 15 22:10:31.954138 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:31.954108    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5873c6d7-792c-41ac-b353-8cccb5f17a9b/kube-rbac-proxy-metric/0.log"
Sep 15 22:10:31.981615 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:31.981530    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5873c6d7-792c-41ac-b353-8cccb5f17a9b/prom-label-proxy/0.log"
Sep 15 22:10:32.011388 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.011340    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5873c6d7-792c-41ac-b353-8cccb5f17a9b/init-config-reloader/0.log"
Sep 15 22:10:32.128069 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.127992    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-sfg9r_7a78378c-9922-4f31-be12-0ad758862a48/kube-state-metrics/0.log"
Sep 15 22:10:32.164021 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.163980    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-sfg9r_7a78378c-9922-4f31-be12-0ad758862a48/kube-rbac-proxy-main/0.log"
Sep 15 22:10:32.202005 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.201978    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-sfg9r_7a78378c-9922-4f31-be12-0ad758862a48/kube-rbac-proxy-self/0.log"
Sep 15 22:10:32.236670 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.236564    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6fb596986f-np7bm_e0e2f419-a8ae-466a-8d60-9c2c7805c1b5/metrics-server/0.log"
Sep 15 22:10:32.276220 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.276180    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-zs2l2_563da805-dd74-457e-81e4-b82796a10106/monitoring-plugin/0.log"
Sep 15 22:10:32.339062 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.339006    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q7wqq_4f0338db-db65-48e6-b35c-e5d7e7bcbb26/node-exporter/0.log"
Sep 15 22:10:32.368540 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.368510    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q7wqq_4f0338db-db65-48e6-b35c-e5d7e7bcbb26/kube-rbac-proxy/0.log"
Sep 15 22:10:32.397334 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.397308    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q7wqq_4f0338db-db65-48e6-b35c-e5d7e7bcbb26/init-textfile/0.log"
Sep 15 22:10:32.702054 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.702001    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-2ws2m_e9ca90f7-2961-4e2d-9345-ec5585c65079/kube-rbac-proxy-main/0.log"
Sep 15 22:10:32.731815 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.731731    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-2ws2m_e9ca90f7-2961-4e2d-9345-ec5585c65079/kube-rbac-proxy-self/0.log"
Sep 15 22:10:32.760819 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.760787    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-2ws2m_e9ca90f7-2961-4e2d-9345-ec5585c65079/openshift-state-metrics/0.log"
Sep 15 22:10:32.809205 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.809130    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a5675c14-383b-4a3a-ad51-5481b7d38c06/prometheus/0.log"
Sep 15 22:10:32.837949 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.837921    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a5675c14-383b-4a3a-ad51-5481b7d38c06/config-reloader/0.log"
Sep 15 22:10:32.863369 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.863343    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a5675c14-383b-4a3a-ad51-5481b7d38c06/thanos-sidecar/0.log"
Sep 15 22:10:32.892209 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.892169    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a5675c14-383b-4a3a-ad51-5481b7d38c06/kube-rbac-proxy-web/0.log"
Sep 15 22:10:32.919861 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.919817    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a5675c14-383b-4a3a-ad51-5481b7d38c06/kube-rbac-proxy/0.log"
Sep 15 22:10:32.948295 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.948263    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a5675c14-383b-4a3a-ad51-5481b7d38c06/kube-rbac-proxy-thanos/0.log"
Sep 15 22:10:32.976586 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:32.976540    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a5675c14-383b-4a3a-ad51-5481b7d38c06/init-config-reloader/0.log"
Sep 15 22:10:33.011119 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:33.011083    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-5bgzn_282361ed-7a1b-459a-adc2-276fdc8688a2/prometheus-operator/0.log"
Sep 15 22:10:33.036133 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:33.036100    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-5bgzn_282361ed-7a1b-459a-adc2-276fdc8688a2/kube-rbac-proxy/0.log"
Sep 15 22:10:33.104803 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:33.104745    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-695955f48-mh7hp_3ce56fc8-7775-4792-ade7-ed91b0c31571/telemeter-client/0.log"
Sep 15 22:10:33.132420 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:33.132396    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-695955f48-mh7hp_3ce56fc8-7775-4792-ade7-ed91b0c31571/reload/0.log"
Sep 15 22:10:33.160112 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:33.160085    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-695955f48-mh7hp_3ce56fc8-7775-4792-ade7-ed91b0c31571/kube-rbac-proxy/0.log"
Sep 15 22:10:33.192399 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:33.192364    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-865dc86548-frw4b_6baef592-84e4-408d-8ee2-8638d6003d7f/thanos-query/0.log"
Sep 15 22:10:33.218102 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:33.218075    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-865dc86548-frw4b_6baef592-84e4-408d-8ee2-8638d6003d7f/kube-rbac-proxy-web/0.log"
Sep 15 22:10:33.243217 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:33.243181    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-865dc86548-frw4b_6baef592-84e4-408d-8ee2-8638d6003d7f/kube-rbac-proxy/0.log"
Sep 15 22:10:33.267558 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:33.267533    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-865dc86548-frw4b_6baef592-84e4-408d-8ee2-8638d6003d7f/prom-label-proxy/0.log"
Sep 15 22:10:33.294184 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:33.294157    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-865dc86548-frw4b_6baef592-84e4-408d-8ee2-8638d6003d7f/kube-rbac-proxy-rules/0.log"
Sep 15 22:10:33.321726 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:33.321677    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-865dc86548-frw4b_6baef592-84e4-408d-8ee2-8638d6003d7f/kube-rbac-proxy-metrics/0.log"
Sep 15 22:10:35.591619 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.591570    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"]
Sep 15 22:10:35.594931 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.594906    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:35.605884 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.605862    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"]
Sep 15 22:10:35.695504 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.695467    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fda21d77-0270-47f0-b9f7-2750361785b5-proc\") pod \"perf-node-gather-daemonset-s7n5r\" (UID: \"fda21d77-0270-47f0-b9f7-2750361785b5\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:35.695791 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.695768    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fda21d77-0270-47f0-b9f7-2750361785b5-podres\") pod \"perf-node-gather-daemonset-s7n5r\" (UID: \"fda21d77-0270-47f0-b9f7-2750361785b5\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:35.695921 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.695907    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fda21d77-0270-47f0-b9f7-2750361785b5-sys\") pod \"perf-node-gather-daemonset-s7n5r\" (UID: \"fda21d77-0270-47f0-b9f7-2750361785b5\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:35.696040 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.696026    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fda21d77-0270-47f0-b9f7-2750361785b5-lib-modules\") pod \"perf-node-gather-daemonset-s7n5r\" (UID: \"fda21d77-0270-47f0-b9f7-2750361785b5\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:35.696166 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.696153    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7z98\" (UniqueName: \"kubernetes.io/projected/fda21d77-0270-47f0-b9f7-2750361785b5-kube-api-access-f7z98\") pod \"perf-node-gather-daemonset-s7n5r\" (UID: \"fda21d77-0270-47f0-b9f7-2750361785b5\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:35.797660 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.797622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fda21d77-0270-47f0-b9f7-2750361785b5-proc\") pod \"perf-node-gather-daemonset-s7n5r\" (UID: \"fda21d77-0270-47f0-b9f7-2750361785b5\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:35.797978 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.797789    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/fda21d77-0270-47f0-b9f7-2750361785b5-proc\") pod \"perf-node-gather-daemonset-s7n5r\" (UID: \"fda21d77-0270-47f0-b9f7-2750361785b5\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:35.798114 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.798092    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fda21d77-0270-47f0-b9f7-2750361785b5-podres\") pod \"perf-node-gather-daemonset-s7n5r\" (UID: \"fda21d77-0270-47f0-b9f7-2750361785b5\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:35.798239 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.798225    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fda21d77-0270-47f0-b9f7-2750361785b5-sys\") pod \"perf-node-gather-daemonset-s7n5r\" (UID: \"fda21d77-0270-47f0-b9f7-2750361785b5\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:35.798422 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.798408    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fda21d77-0270-47f0-b9f7-2750361785b5-lib-modules\") pod \"perf-node-gather-daemonset-s7n5r\" (UID: \"fda21d77-0270-47f0-b9f7-2750361785b5\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:35.798681 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.798660    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7z98\" (UniqueName: \"kubernetes.io/projected/fda21d77-0270-47f0-b9f7-2750361785b5-kube-api-access-f7z98\") pod \"perf-node-gather-daemonset-s7n5r\" (UID: \"fda21d77-0270-47f0-b9f7-2750361785b5\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:35.799160 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.798122    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/fda21d77-0270-47f0-b9f7-2750361785b5-podres\") pod \"perf-node-gather-daemonset-s7n5r\" (UID: \"fda21d77-0270-47f0-b9f7-2750361785b5\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:35.799301 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.798585    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fda21d77-0270-47f0-b9f7-2750361785b5-lib-modules\") pod \"perf-node-gather-daemonset-s7n5r\" (UID: \"fda21d77-0270-47f0-b9f7-2750361785b5\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:35.799406 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.798360    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fda21d77-0270-47f0-b9f7-2750361785b5-sys\") pod \"perf-node-gather-daemonset-s7n5r\" (UID: \"fda21d77-0270-47f0-b9f7-2750361785b5\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:35.808228 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.808209    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7z98\" (UniqueName: \"kubernetes.io/projected/fda21d77-0270-47f0-b9f7-2750361785b5-kube-api-access-f7z98\") pod \"perf-node-gather-daemonset-s7n5r\" (UID: \"fda21d77-0270-47f0-b9f7-2750361785b5\") " pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:35.910706 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:35.910673    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:36.213782 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:36.213733    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"]
Sep 15 22:10:36.215956 ip-10-0-135-148 kubenswrapper[2575]: W0915 22:10:36.215919    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfda21d77_0270_47f0_b9f7_2750361785b5.slice/crio-b45bf7484473aa5a4556394e72154f27bb99788b38b5a762dc3eaed0ff060c0b WatchSource:0}: Error finding container b45bf7484473aa5a4556394e72154f27bb99788b38b5a762dc3eaed0ff060c0b: Status 404 returned error can't find the container with id b45bf7484473aa5a4556394e72154f27bb99788b38b5a762dc3eaed0ff060c0b
Sep 15 22:10:36.591530 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:36.591443    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r" event={"ID":"fda21d77-0270-47f0-b9f7-2750361785b5","Type":"ContainerStarted","Data":"fa8113bfeae9773d7300ab9ccb6b564a7c423cb9a011a6c383f1a9b7547b58f1"}
Sep 15 22:10:36.592480 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:36.592449    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r" event={"ID":"fda21d77-0270-47f0-b9f7-2750361785b5","Type":"ContainerStarted","Data":"b45bf7484473aa5a4556394e72154f27bb99788b38b5a762dc3eaed0ff060c0b"}
Sep 15 22:10:36.592934 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:36.592498    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:36.608006 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:36.607946    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r" podStartSLOduration=1.60792709 podStartE2EDuration="1.60792709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 22:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 22:10:36.607253086 +0000 UTC m=+2674.419261557" watchObservedRunningTime="2026-09-15 22:10:36.60792709 +0000 UTC m=+2674.419935535"
Sep 15 22:10:37.819878 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:37.819849    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sqhn2_dbe46385-fe7f-418f-abd9-107a9da93dba/dns/0.log"
Sep 15 22:10:37.847556 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:37.847510    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sqhn2_dbe46385-fe7f-418f-abd9-107a9da93dba/kube-rbac-proxy/0.log"
Sep 15 22:10:37.952643 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:37.952614    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rwmx9_895233b3-517c-4b44-9dc1-02cb2e4ff8ac/dns-node-resolver/0.log"
Sep 15 22:10:38.597203 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:38.597172    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wwhph_6b3f47d0-ec5b-4699-91a9-160ea99f70fc/node-ca/0.log"
Sep 15 22:10:40.435659 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:40.435631    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d78dff5cc-c254b_993a3b58-915c-48cf-860e-0695b63089b4/kube-auth-proxy/0.log"
Sep 15 22:10:40.537417 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:40.537382    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xx5vp_2e58dd44-0283-4acd-a0c1-3984a3451a52/payload-pre-processing/0.log"
Sep 15 22:10:41.593769 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:41.593730    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k2lzl_18d9cbb1-4fad-45f1-ae92-4125ca00a324/serve-healthcheck-canary/0.log"
Sep 15 22:10:42.124370 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:42.124341    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bqkb9_b2de7b26-21f1-475f-b5e4-fe26b4032240/kube-rbac-proxy/0.log"
Sep 15 22:10:42.160168 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:42.160136    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bqkb9_b2de7b26-21f1-475f-b5e4-fe26b4032240/exporter/0.log"
Sep 15 22:10:42.208466 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:42.208420    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bqkb9_b2de7b26-21f1-475f-b5e4-fe26b4032240/extractor/0.log"
Sep 15 22:10:42.616935 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:42.616905    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-lgf4x/perf-node-gather-daemonset-s7n5r"
Sep 15 22:10:45.790033 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:45.790003    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77fddcc9c6-qzxzg_8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855/manager/0.log"
Sep 15 22:10:45.815711 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:45.815689    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-77fddcc9c6-qzxzg_8ca1bf6f-b8cd-4cc2-a0fd-b8a2f067d855/copy-manifests/0.log"
Sep 15 22:10:46.521682 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:46.521651    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d78dff5cc-c254b_993a3b58-915c-48cf-860e-0695b63089b4/kube-auth-proxy/0.log"
Sep 15 22:10:46.608810 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:46.608776    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xx5vp_2e58dd44-0283-4acd-a0c1-3984a3451a52/payload-pre-processing/0.log"
Sep 15 22:10:47.037001 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:47.036967    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5d697dcf85-gtz98_eac363c9-9f81-4e65-abb6-3ed190030f39/manager/0.log"
Sep 15 22:10:49.132285 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:49.132258    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d78dff5cc-c254b_993a3b58-915c-48cf-860e-0695b63089b4/kube-auth-proxy/0.log"
Sep 15 22:10:49.210527 ip-10-0-135-148 kubenswrapper[2575]: I0915 22:10:49.210483    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-xx5vp_2e58dd44-0283-4acd-a0c1-3984a3451a52/payload-pre-processing/0.log"