Sep 11 18:13:12.375930 ip-10-0-129-166 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 18:13:12.375942 ip-10-0-129-166 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 18:13:12.375951 ip-10-0-129-166 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 18:13:12.376245 ip-10-0-129-166 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 18:13:23.532006 ip-10-0-129-166 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 18:13:23.532025 ip-10-0-129-166 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot ad9540dea35e4d39adafc8faf9afbbb8 --
Sep 11 18:15:23.145749 ip-10-0-129-166 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 18:15:23.519141 ip-10-0-129-166 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:15:23.519141 ip-10-0-129-166 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 18:15:23.519141 ip-10-0-129-166 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:15:23.519141 ip-10-0-129-166 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 18:15:23.519141 ip-10-0-129-166 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:15:23.521485 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.521390    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 18:15:23.524528 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524513    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:23.524528 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524528    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524532    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524535    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524538    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524541    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524544    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524548    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524552    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524554    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524557    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524559    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524562    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524564    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524567    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524569    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524572    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524574    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524577    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524579    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:23.524594 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524582    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524585    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524587    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524595    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524597    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524600    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524603    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524605    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524608    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524610    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524614    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524616    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524618    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524621    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524624    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524626    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524629    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524631    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524634    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524637    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:23.525060 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524639    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524643    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524646    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524649    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524652    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524655    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524657    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524660    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524663    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524666    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524670    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524672    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524675    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524678    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524681    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524684    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524687    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524690    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524693    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:23.525566 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524696    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524698    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524700    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524703    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524705    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524708    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524710    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524713    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524715    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524718    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524720    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524723    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524725    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524728    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524730    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524733    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524735    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524738    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524740    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524742    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:23.526040 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524745    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524747    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524750    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524752    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524755    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524757    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524761    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524787    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524791    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524794    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524797    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524800    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524802    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524805    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524807    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524810    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524812    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524815    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524817    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.524820    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:23.526573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525498    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525505    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525508    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525511    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525513    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525516    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525519    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525522    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525524    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525527    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525529    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525533    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525536    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525538    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525541    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525544    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525547    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525550    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525552    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525555    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:23.527093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525557    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525560    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525563    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525565    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525568    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525570    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525573    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525576    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525578    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525581    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525585    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525589    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525592    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525594    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525597    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525599    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525601    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525604    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525607    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:23.527600 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525610    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525612    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525616    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525621    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525624    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525627    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525630    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525633    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525636    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525639    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525642    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525645    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525647    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525650    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525653    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525655    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525658    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525660    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525663    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:23.528063 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525665    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525668    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525670    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525673    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525675    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525677    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525681    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525683    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525686    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525688    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525692    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525695    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525698    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525701    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525704    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525707    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525709    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525711    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525714    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525716    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:23.528558 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525719    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525721    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525724    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525727    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525730    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525732    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525734    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525737    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525739    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525742    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525744    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525746    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525749    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525751    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525753    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525756    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525758    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525761    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525763    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525766    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:23.529041 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.525769    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526151    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526156    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526159    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526162    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526165    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526168    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526171    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526174    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526177    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526180    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526182    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526185    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526188    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526190    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526194    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526198    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526201    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526204    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526206    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:23.529544 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526209    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526211    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526214    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526217    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526221    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526224    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526227    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526230    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526232    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526235    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526237    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526239    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526242    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526245    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526247    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526250    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526253    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526255    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526258    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:23.530024 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526261    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526263    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526265    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526268    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526270    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526273    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526276    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526278    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526281    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526283    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526286    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526288    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526290    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526292    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526295    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526297    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526300    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526302    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526304    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:23.530502 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526307    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526309    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526312    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526314    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526316    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526319    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526321    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526324    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526326    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526329    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526331    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526333    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526337    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526340    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526342    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526344    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526347    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526349    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526352    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526357    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:23.530984 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526361    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526364    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526367    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526370    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526372    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526375    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526377    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526379    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526382    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526385    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526388    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526390    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526392    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526395    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526397    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526399    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526402    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526404    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526407    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:23.531489 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526409    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526412    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.526414    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526494    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526527    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526535    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526539    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526544    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526547    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526551    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526555    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526558    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526562    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526566    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526569    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526572    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526575    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526578    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526581    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526584    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526587    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526591    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526594    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 18:15:23.532018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526597    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526600    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526604    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526607    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526610    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526613    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526616    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526620    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526623    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526626    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526629    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526632    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526636    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526639    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526641    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526644    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526647    2581 flags.go:64] FLAG: --enable-server="true"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526651    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526655    2581 flags.go:64] FLAG: --event-burst="100"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526658    2581 flags.go:64] FLAG: --event-qps="50"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526661    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526664    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526667    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526671    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 18:15:23.532597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526673    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526676    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526679    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526682    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526684    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526688    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526691    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526693    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526696    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526699    2581 flags.go:64] FLAG: --feature-gates=""
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526702    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526705    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526708    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526711    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526714    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526717    2581 flags.go:64] FLAG: --help="false"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526721    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-129-166.ec2.internal"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526724    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526727    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526730    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526733    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526736    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526739    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 18:15:23.533168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526742    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526744    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526747    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526750    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526753    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526756    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526759    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526762    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526765    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526768    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526771    2581 flags.go:64] FLAG: --lock-file=""
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526773    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526776    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526779    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526784    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526787    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526789    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526792    2581 flags.go:64] FLAG: --logging-format="text"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526795    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526798    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526800    2581 flags.go:64] FLAG: --manifest-url=""
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526803    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526807    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526810    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526814    2581 flags.go:64] FLAG: --max-pods="110"
Sep 11 18:15:23.533766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526819    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526821    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526824    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526827    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526830    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526833    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526836    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526843    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526846    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526849    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526851    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526855    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526860    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526862    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526866    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526869    2581 flags.go:64] FLAG: --port="10250"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526872    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526874    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04fb87ec9ff0c1569"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526877    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526880    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526883    2581 flags.go:64] FLAG: --register-node="true"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526886    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526890    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526892    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 11 18:15:23.534408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526895    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526898    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526901    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526904    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526907    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526909    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526912    2581 flags.go:64] FLAG: --runonce="false"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526915    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526919    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526922    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526925    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526927    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526930    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526933    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526936    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526939    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526942    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526945    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526948    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526951    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526954    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526956    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526962    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526965    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526967    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 18:15:23.535008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526971    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526974    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526977    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526980    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526982    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526985    2581 flags.go:64] FLAG: --v="2"
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526989    2581 flags.go:64] FLAG: --version="false"
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526992    2581 flags.go:64] FLAG: --vmodule=""
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526996    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.526999    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527112    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527115    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527118    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527121    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527123    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527127    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527130    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527133    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527135    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527138    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527140    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527143    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:23.535680 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527145    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527148    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527150    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527155    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527159    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527161    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527164    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527167    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527169    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527171    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527174    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527177    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527179    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527182    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527184    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527187    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527189    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527193    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527197    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527199    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:23.536318 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527202    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527205    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527208    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527212    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527215    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527221    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527224    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527227    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527229    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527232    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527235    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527237    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527240    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527242    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527244    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527248    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527251    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527254    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527256    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527260    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:23.536820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527262    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527265    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527267    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527270    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527272    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527275    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527277    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527280    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527284    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527288    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527291    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527293    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527296    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527299    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527302    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527304    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527306    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527310    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527313    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:23.537309 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527316    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527319    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527323    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527326    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527329    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527331    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527334    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527336    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527340    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527343    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527346    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527348    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527351    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527354    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527357    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527359    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527362    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527364    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527367    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:23.537781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527369    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:23.538243 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527372    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:23.538243 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527374    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:23.538243 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527376    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:23.538243 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527379    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:23.538243 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527381    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:23.538243 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527384    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:23.538243 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527386    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:23.538243 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.527389    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:23.538243 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.528064    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:15:23.538243 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.535421    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 18:15:23.538243 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.535435    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 18:15:23.538243 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.535454    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 18:15:23.538243 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.535460    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 18:15:23.538243 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535536    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:23.538243 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535541    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535545    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535548    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535552    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535555    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535558    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535560    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535563    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535566    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535569    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535571    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535575    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535577    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535580    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535582    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535586    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535588    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535591    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535593    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535596    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:23.538677 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535599    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535601    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535604    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535606    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535610    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535614    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535617    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535621    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535624    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535627    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535629    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535632    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535634    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535637    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535639    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535642    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535644    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535647    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535649    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535652    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:23.539221 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535654    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535657    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535659    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535662    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535665    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535667    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535670    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535672    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535675    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535678    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535680    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535683    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535686    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535688    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535691    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535693    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535696    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535699    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535701    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:23.540007 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535703    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535706    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535709    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535712    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535714    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535716    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535719    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535721    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535724    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535726    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535729    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535731    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535734    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535737    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535740    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535744    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535746    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535749    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535752    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:23.540733 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535754    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535756    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535759    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535761    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535764    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535766    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535769    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535771    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535774    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535776    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535780    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535783    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535786    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535789    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535792    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535795    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535798    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535800    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535803    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:23.541303 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535805    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:23.541793 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.535810    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:15:23.541793 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535932    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:23.541793 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535937    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:23.541793 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535940    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:23.541793 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535943    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:23.541793 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535946    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:23.541793 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535949    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:23.541793 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535952    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:23.541793 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535955    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:23.541793 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535957    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:23.541793 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535960    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:23.541793 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535962    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:23.541793 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535966    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:23.541793 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535969    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:23.541793 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535972    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535974    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535977    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535979    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535982    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535985    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535987    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535990    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535992    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535995    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.535998    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536000    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536003    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536005    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536008    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536010    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536013    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536015    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536018    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536020    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:23.542207 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536023    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536025    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536027    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536030    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536032    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536035    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536037    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536040    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536042    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536044    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536047    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536050    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536053    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536055    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536058    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536060    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536063    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536065    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536068    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536071    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:23.542705 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536073    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536076    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536078    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536081    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536083    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536086    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536088    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536091    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536093    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536096    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536098    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536100    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536103    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536105    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536108    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536111    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536113    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536115    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536118    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536120    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:23.543238 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536123    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536125    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536128    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536130    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536133    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536137    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536140    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536143    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536146    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536148    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536151    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536154    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536157    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536160    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536162    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536165    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536167    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536170    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536172    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:23.543755 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536175    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:23.544226 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536177    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:23.544226 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536180    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:23.544226 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536183    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:23.544226 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536187    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:23.544226 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536189    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:23.544226 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:23.536193    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:23.544226 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.536198    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:15:23.544226 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.536296    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 18:15:23.544226 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.538448    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 18:15:23.544226 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.539235    2581 server.go:1019] "Starting client certificate rotation"
Sep 11 18:15:23.544226 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.539352    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 18:15:23.544226 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.539411    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 18:15:23.544226 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.541449    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 18:15:23.546829 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.546799    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 18:15:23.556890 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.556872    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 11 18:15:23.560104 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.560088    2581 log.go:25] "Validated CRI v1 image API"
Sep 11 18:15:23.560969 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.560956    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 18:15:23.563127 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.563109    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c687e1ea-1273-427a-aa16-fe5c6bb7652f:/dev/nvme0n1p4 ee055863-636f-411c-9061-b6f3ea8097c6:/dev/nvme0n1p3]
Sep 11 18:15:23.563190 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.563126    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 18:15:23.568775 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.567886    2581 manager.go:216] Machine: {Timestamp:2026-09-11 18:15:23.566688433 +0000 UTC m=+0.307815267 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3013891 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c6af8ac9d607f97460a18d31b4106 SystemUUID:ec2c6af8-ac9d-607f-9746-0a18d31b4106 BootID:ad9540de-a35e-4d39-adaf-c8faf9afbbb8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ec:39:b8:8d:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ec:39:b8:8d:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:96:3b:56:20:45 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 18:15:23.568775 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.568762    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 18:15:23.568906 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.568841    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 18:15:23.569248 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.569233    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 18:15:23.572536 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.572503    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 18:15:23.572690 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.572540    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-166.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 18:15:23.573305 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.573296    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 18:15:23.573335 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.573307    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 18:15:23.573335 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.573315    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 18:15:23.574054 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.574044    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 18:15:23.575143 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.575134    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 18:15:23.579006 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.578995    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 18:15:23.579150 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.579140    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 18:15:23.579194 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.579170    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 18:15:23.579194 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.579179    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 18:15:23.579194 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.579189    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 18:15:23.579309 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.579198    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 11 18:15:23.579309 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.579207    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 18:15:23.579568 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.579557    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 18:15:23.579615 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.579577    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 18:15:23.581545 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.581526    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 18:15:23.584641 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.584626    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 18:15:23.584705 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.584656    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 18:15:23.586209 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.586197    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 18:15:23.586209 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.586212    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 18:15:23.586301 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.586226    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 18:15:23.586301 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.586231    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 18:15:23.586301 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.586240    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 18:15:23.586301 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.586245    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 18:15:23.586301 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.586258    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 18:15:23.586301 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.586263    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 18:15:23.586301 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.586269    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 18:15:23.586301 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.586274    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 18:15:23.586301 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.586286    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 18:15:23.586301 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.586291    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 18:15:23.586695 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.586314    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 18:15:23.586695 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.586320    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 18:15:23.590150 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.590136    2581 server.go:1268] "Started kubelet"
Sep 11 18:15:23.590211 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.590175    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 18:15:23.590281 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.590221    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 18:15:23.590400 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.590315    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 18:15:23.590906 ip-10-0-129-166 systemd[1]: Started Kubernetes Kubelet.
Sep 11 18:15:23.593981 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.593948    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-166.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 18:15:23.593981 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.593961    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 18:15:23.594135 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.594120    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-166.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 18:15:23.594291 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.594269    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 18:15:23.595834 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.595810    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 18:15:23.595965 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.595951    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 18:15:23.598282 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.597445    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-166.ec2.internal.18d456ec0048b7f4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-166.ec2.internal,UID:ip-10-0-129-166.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-166.ec2.internal,},FirstTimestamp:2026-09-11 18:15:23.589965812 +0000 UTC m=+0.331092645,LastTimestamp:2026-09-11 18:15:23.589965812 +0000 UTC m=+0.331092645,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-166.ec2.internal,}"
Sep 11 18:15:23.602403 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.602383    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 18:15:23.602931 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.602914    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 18:15:23.603013 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.602985    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-166.ec2.internal\" not found"
Sep 11 18:15:23.603605 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.603585    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 18:15:23.603605 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.603588    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 18:15:23.603713 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.603615    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 18:15:23.603861 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.603813    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 18:15:23.603902 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.603862    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 18:15:23.605075 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.605059    2581 factory.go:55] Registering systemd factory
Sep 11 18:15:23.605075 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.605076    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 11 18:15:23.605284 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.605264    2581 factory.go:153] Registering CRI-O factory
Sep 11 18:15:23.605284 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.605284    2581 factory.go:223] Registration of the crio container factory successfully
Sep 11 18:15:23.605399 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.605321    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 18:15:23.605399 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.605339    2581 factory.go:103] Registering Raw factory
Sep 11 18:15:23.605399 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.605349    2581 manager.go:1196] Started watching for new ooms in manager
Sep 11 18:15:23.605530 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.605406    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cm42k"
Sep 11 18:15:23.605843 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.605706    2581 manager.go:349] Starting recovery of all containers
Sep 11 18:15:23.613672 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.613650    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cm42k"
Sep 11 18:15:23.614260 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.614236    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-166.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 18:15:23.614698 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.614674    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 18:15:23.617029 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.617013    2581 manager.go:354] Recovery completed
Sep 11 18:15:23.621411 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.621399    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:23.623094 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.623070    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:23.623170 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.623109    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:23.623170 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.623122    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:23.623669 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.623653    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 18:15:23.623669 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.623667    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 18:15:23.623669 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.623670    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:23.623809 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.623680    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 18:15:23.624252 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.624237    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:23.624302 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.624262    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:23.624302 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.624273    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:23.624937 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.624927    2581 policy_none.go:49] "None policy: Start"
Sep 11 18:15:23.624979 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.624942    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 18:15:23.624979 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.624952    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 18:15:23.625957 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.625947    2581 policy_none.go:47] "Start"
Sep 11 18:15:23.663138 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.663021    2581 manager.go:337] "Starting Device Plugin manager"
Sep 11 18:15:23.684597 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.663158    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 18:15:23.684597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.663168    2581 server.go:87] "Starting device plugin registration server"
Sep 11 18:15:23.684597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.663384    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 18:15:23.684597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.663393    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 18:15:23.684597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.663503    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 18:15:23.684597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.663581    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 18:15:23.684597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.663590    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 18:15:23.684597 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.664020    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 18:15:23.684597 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.664069    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-166.ec2.internal\" not found"
Sep 11 18:15:23.755880 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.755852    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 18:15:23.757127 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.757111    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 18:15:23.757216 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.757156    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 18:15:23.757216 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.757175    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 18:15:23.757216 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.757207    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 18:15:23.759966 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.759945    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:23.764124 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.764112    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:23.764901 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.764888    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:23.764994 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.764919    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:23.764994 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.764934    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:23.764994 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.764963    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-166.ec2.internal"
Sep 11 18:15:23.774216 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.774174    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-166.ec2.internal"
Sep 11 18:15:23.774216 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.774194    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-166.ec2.internal\": node \"ip-10-0-129-166.ec2.internal\" not found"
Sep 11 18:15:23.790970 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.790947    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-166.ec2.internal\" not found"
Sep 11 18:15:23.858217 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.858197    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-166.ec2.internal"]
Sep 11 18:15:23.858297 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.858257    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:23.859844 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.859827    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:23.859935 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.859858    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:23.859935 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.859870    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:23.861007 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.860995    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:23.861140 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.861127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal"
Sep 11 18:15:23.861176 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.861158    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:23.861849 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.861827    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:23.861952 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.861851    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:23.861952 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.861860    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:23.861952 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.861864    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:23.861952 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.861888    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:23.861952 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.861897    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:23.862946 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.862933    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-166.ec2.internal"
Sep 11 18:15:23.862993 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.862955    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:23.863625 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.863609    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:23.863701 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.863636    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:23.863701 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.863649    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:23.884005 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.883986    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-166.ec2.internal\" not found" node="ip-10-0-129-166.ec2.internal"
Sep 11 18:15:23.888402 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.888387    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-166.ec2.internal\" not found" node="ip-10-0-129-166.ec2.internal"
Sep 11 18:15:23.891448 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.891430    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-166.ec2.internal\" not found"
Sep 11 18:15:23.906065 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.906043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e76cbbfe0210854c66f75b5edb339223-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal\" (UID: \"e76cbbfe0210854c66f75b5edb339223\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal"
Sep 11 18:15:23.906145 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.906068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e76cbbfe0210854c66f75b5edb339223-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal\" (UID: \"e76cbbfe0210854c66f75b5edb339223\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal"
Sep 11 18:15:23.906145 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:23.906084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1fccd7d8530b9bbb746ecca8d0ec2393-config\") pod \"kube-apiserver-proxy-ip-10-0-129-166.ec2.internal\" (UID: \"1fccd7d8530b9bbb746ecca8d0ec2393\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-166.ec2.internal"
Sep 11 18:15:23.991749 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:23.991721    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-166.ec2.internal\" not found"
Sep 11 18:15:24.007049 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.007031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1fccd7d8530b9bbb746ecca8d0ec2393-config\") pod \"kube-apiserver-proxy-ip-10-0-129-166.ec2.internal\" (UID: \"1fccd7d8530b9bbb746ecca8d0ec2393\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-166.ec2.internal"
Sep 11 18:15:24.007125 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.007052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e76cbbfe0210854c66f75b5edb339223-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal\" (UID: \"e76cbbfe0210854c66f75b5edb339223\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal"
Sep 11 18:15:24.007125 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.007069    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e76cbbfe0210854c66f75b5edb339223-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal\" (UID: \"e76cbbfe0210854c66f75b5edb339223\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal"
Sep 11 18:15:24.007222 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.007132    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e76cbbfe0210854c66f75b5edb339223-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal\" (UID: \"e76cbbfe0210854c66f75b5edb339223\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal"
Sep 11 18:15:24.007222 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.007140    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1fccd7d8530b9bbb746ecca8d0ec2393-config\") pod \"kube-apiserver-proxy-ip-10-0-129-166.ec2.internal\" (UID: \"1fccd7d8530b9bbb746ecca8d0ec2393\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-166.ec2.internal"
Sep 11 18:15:24.007222 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.007151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e76cbbfe0210854c66f75b5edb339223-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal\" (UID: \"e76cbbfe0210854c66f75b5edb339223\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal"
Sep 11 18:15:24.092842 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:24.092773    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-166.ec2.internal\" not found"
Sep 11 18:15:24.187276 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.187238    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal"
Sep 11 18:15:24.192458 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.192442    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-166.ec2.internal"
Sep 11 18:15:24.193534 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:24.193511    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-166.ec2.internal\" not found"
Sep 11 18:15:24.293962 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:24.293919    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-166.ec2.internal\" not found"
Sep 11 18:15:24.394445 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:24.394387    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-166.ec2.internal\" not found"
Sep 11 18:15:24.495092 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:24.495057    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-166.ec2.internal\" not found"
Sep 11 18:15:24.539616 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.539593    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 18:15:24.539995 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.539710    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 18:15:24.594139 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.594118    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:24.603170 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.603151    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal"
Sep 11 18:15:24.603170 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.603161    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 18:15:24.613749 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.613728    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 18:15:24.615238 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.615221    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 18:15:24.616197 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.616184    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-166.ec2.internal"
Sep 11 18:15:24.616640 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.616613    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 18:10:24 +0000 UTC" deadline="2028-05-05 16:21:12.177418849 +0000 UTC"
Sep 11 18:15:24.616690 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.616639    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14446h5m47.560781886s"
Sep 11 18:15:24.621066 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.621054    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:24.622635 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.622623    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 18:15:24.752383 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.752361    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bjgbb"
Sep 11 18:15:24.760468 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.760450    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bjgbb"
Sep 11 18:15:24.836646 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:24.836620    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fccd7d8530b9bbb746ecca8d0ec2393.slice/crio-b247c904ac13a9091e9927cb3f3428fd54c20a85a9713b4fe7a6cbd4bb41da56 WatchSource:0}: Error finding container b247c904ac13a9091e9927cb3f3428fd54c20a85a9713b4fe7a6cbd4bb41da56: Status 404 returned error can't find the container with id b247c904ac13a9091e9927cb3f3428fd54c20a85a9713b4fe7a6cbd4bb41da56
Sep 11 18:15:24.836843 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:24.836823    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode76cbbfe0210854c66f75b5edb339223.slice/crio-6b1dba8dd6d0d8aaea01d9888862fe306075356a1da63a2eaebfb5d312657810 WatchSource:0}: Error finding container 6b1dba8dd6d0d8aaea01d9888862fe306075356a1da63a2eaebfb5d312657810: Status 404 returned error can't find the container with id 6b1dba8dd6d0d8aaea01d9888862fe306075356a1da63a2eaebfb5d312657810
Sep 11 18:15:24.840147 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:24.840133    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:15:25.028688 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.028663    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:25.580582 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.580543    2581 apiserver.go:52] "Watching apiserver"
Sep 11 18:15:25.587849 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.587822    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 18:15:25.589959 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.589928    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-fxtlq","kube-system/global-pull-secret-syncer-9rs25","kube-system/kube-apiserver-proxy-ip-10-0-129-166.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg","openshift-dns/node-resolver-b9dfv","openshift-network-operator/iptables-alerter-q8nvm","openshift-ovn-kubernetes/ovnkube-node-mx4jb","kube-system/konnectivity-agent-zdcg9","openshift-cluster-node-tuning-operator/tuned-lw4pm","openshift-image-registry/node-ca-xvv2h","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal","openshift-multus/multus-additional-cni-plugins-p6fm6","openshift-multus/multus-sz67x","openshift-multus/network-metrics-daemon-48sl5"]
Sep 11 18:15:25.591918 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.591899    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.594377 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.594113    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.595809 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.595131    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 18:15:25.595809 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.595147    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-znxc9\""
Sep 11 18:15:25.595809 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.595219    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 18:15:25.595809 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.595409    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 18:15:25.595809 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.595503    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 18:15:25.595809 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.595622    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 18:15:25.595809 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.595628    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 18:15:25.595809 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.595781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:25.596179 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:25.595878    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:25.597586 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.596887    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 18:15:25.597586 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.597393    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 18:15:25.597586 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.597498    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 18:15:25.597794 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.597728    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-927s8\""
Sep 11 18:15:25.597794 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.597783    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 18:15:25.598449 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.598163    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.600241 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.599971    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b9dfv"
Sep 11 18:15:25.600241 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.600114    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 18:15:25.600499 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.600478    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v54qz\""
Sep 11 18:15:25.600653 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.600575    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 18:15:25.600733 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.600706    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 18:15:25.601739 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.601722    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q8nvm"
Sep 11 18:15:25.602465 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.602443    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 18:15:25.602990 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.602960    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 18:15:25.603108 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.603045    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9rcc8\""
Sep 11 18:15:25.603108 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.603067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.604003 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.603981    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 18:15:25.604665 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.604640    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-kqztf\""
Sep 11 18:15:25.605024 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.605004    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:15:25.605259 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.605241    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 18:15:25.606720 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.606694    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 18:15:25.606970 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.606952    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fwv28\""
Sep 11 18:15:25.607190 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.607175    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:15:25.608671 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.608653    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:25.608795 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:25.608768    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:25.610778 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.610739    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:25.610946 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:25.610919    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:25.613239 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.613221    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xvv2h"
Sep 11 18:15:25.615382 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.615282    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zdcg9"
Sep 11 18:15:25.615382 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.615373    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 18:15:25.616137 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.615588    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vm7kf\""
Sep 11 18:15:25.616137 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.615648    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 18:15:25.616137 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.615666    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 18:15:25.617300 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.617277    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mnd8f\""
Sep 11 18:15:25.617645 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.617628    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 18:15:25.618148 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618128    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.618340 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618130    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 18:15:25.618422 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-multus-socket-dir-parent\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.618422 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-etc-kubernetes\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.618597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618443    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-systemd-units\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.618597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618466    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-var-lib-openvswitch\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.618597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62fdeac2-769b-411d-b103-9ca91d804f1f-ovnkube-config\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.618597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618556    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62fdeac2-769b-411d-b103-9ca91d804f1f-env-overrides\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.618859 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab698762-88b3-4098-bf05-6718e735177a-multus-daemon-config\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.618859 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8p9n\" (UniqueName: \"kubernetes.io/projected/68330d27-5afb-44d1-96a2-774104ee6c4c-kube-api-access-p8p9n\") pod \"node-resolver-b9dfv\" (UID: \"68330d27-5afb-44d1-96a2-774104ee6c4c\") " pod="openshift-dns/node-resolver-b9dfv"
Sep 11 18:15:25.618859 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-run-systemd\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.618859 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-run-k8s-cni-cncf-io\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.618859 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618731    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-socket-dir\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.618859 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-run-netns\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.618859 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618790    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-device-dir\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.619214 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68330d27-5afb-44d1-96a2-774104ee6c4c-tmp-dir\") pod \"node-resolver-b9dfv\" (UID: \"68330d27-5afb-44d1-96a2-774104ee6c4c\") " pod="openshift-dns/node-resolver-b9dfv"
Sep 11 18:15:25.619214 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618890    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-lib-modules\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.619214 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-hostroot\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.619214 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-run-multus-certs\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.619214 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.619214 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.618988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-etc-selinux\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.619214 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619012    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-tuned\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.619214 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qgq7\" (UniqueName: \"kubernetes.io/projected/7a3a8fa6-2a7b-4d7a-ac23-63babb814575-kube-api-access-6qgq7\") pod \"iptables-alerter-q8nvm\" (UID: \"7a3a8fa6-2a7b-4d7a-ac23-63babb814575\") " pod="openshift-network-operator/iptables-alerter-q8nvm"
Sep 11 18:15:25.619214 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-modprobe-d\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.619214 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619163    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab698762-88b3-4098-bf05-6718e735177a-cni-binary-copy\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.619214 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj7cz\" (UniqueName: \"kubernetes.io/projected/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-kube-api-access-dj7cz\") pod \"network-metrics-daemon-48sl5\" (UID: \"db03c3a4-cfd5-443f-afe5-6e2fad4b0678\") " pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:25.619916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a3a8fa6-2a7b-4d7a-ac23-63babb814575-host-slash\") pod \"iptables-alerter-q8nvm\" (UID: \"7a3a8fa6-2a7b-4d7a-ac23-63babb814575\") " pod="openshift-network-operator/iptables-alerter-q8nvm"
Sep 11 18:15:25.619916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-sysctl-d\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.619916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d282c\" (UniqueName: \"kubernetes.io/projected/1462a6c5-66bd-4a38-ac43-a80933ec590d-kube-api-access-d282c\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.619916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-sysconfig\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.619916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-node-log\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.619916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-cni-netd\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.619916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-system-cni-dir\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.619916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619457    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-var-lib-kubelet\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.619916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxdjs\" (UniqueName: \"kubernetes.io/projected/ab698762-88b3-4098-bf05-6718e735177a-kube-api-access-wxdjs\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.619916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs\") pod \"network-metrics-daemon-48sl5\" (UID: \"db03c3a4-cfd5-443f-afe5-6e2fad4b0678\") " pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:25.619916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-sys-fs\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.619916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619603    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-kubelet\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.619916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-run-openvswitch\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.619916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619729    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-registration-dir\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.619916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-systemd\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.619916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-run\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619805    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62fdeac2-769b-411d-b103-9ca91d804f1f-ovn-node-metrics-cert\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grb7r\" (UniqueName: \"kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r\") pod \"network-check-target-fxtlq\" (UID: \"a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b\") " pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-etc-openvswitch\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619887    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-multus-cni-dir\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-multus-conf-dir\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619945    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/68330d27-5afb-44d1-96a2-774104ee6c4c-hosts-file\") pod \"node-resolver-b9dfv\" (UID: \"68330d27-5afb-44d1-96a2-774104ee6c4c\") " pod="openshift-dns/node-resolver-b9dfv"
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-host\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.619993    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9dl6p\""
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620031    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-kubernetes\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620302    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-run-netns\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-var-lib-cni-multus\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620398    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a3a8fa6-2a7b-4d7a-ac23-63babb814575-iptables-alerter-script\") pod \"iptables-alerter-q8nvm\" (UID: \"7a3a8fa6-2a7b-4d7a-ac23-63babb814575\") " pod="openshift-network-operator/iptables-alerter-q8nvm"
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620429    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-log-socket\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620472    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1462a6c5-66bd-4a38-ac43-a80933ec590d-tmp\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.620883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.621899 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-cni-bin\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.621899 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620570    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62fdeac2-769b-411d-b103-9ca91d804f1f-ovnkube-script-lib\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.621899 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x9nz\" (UniqueName: \"kubernetes.io/projected/62fdeac2-769b-411d-b103-9ca91d804f1f-kube-api-access-5x9nz\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.621899 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-var-lib-cni-bin\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.621899 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-sys\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.621899 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9b6c\" (UniqueName: \"kubernetes.io/projected/8de484a7-3da4-4dd7-82da-46fe04ad4508-kube-api-access-x9b6c\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.621899 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-sysctl-conf\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.621899 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-var-lib-kubelet\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.621899 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-slash\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.621899 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620842    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-cnibin\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.621899 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.620889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-run-ovn\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.705342 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.705316    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 18:15:25.722021 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.721993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a3a8fa6-2a7b-4d7a-ac23-63babb814575-iptables-alerter-script\") pod \"iptables-alerter-q8nvm\" (UID: \"7a3a8fa6-2a7b-4d7a-ac23-63babb814575\") " pod="openshift-network-operator/iptables-alerter-q8nvm"
Sep 11 18:15:25.722144 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-log-socket\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.722144 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722059    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1462a6c5-66bd-4a38-ac43-a80933ec590d-tmp\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.722144 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/608a1286-9f29-437e-b113-0f4cf3015c7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.722314 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.722314 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-cni-bin\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.722314 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722188    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-log-socket\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.722314 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62fdeac2-769b-411d-b103-9ca91d804f1f-ovnkube-script-lib\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.722314 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722246    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-run-ovn-kubernetes\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.722314 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722261    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-cni-bin\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.722314 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x9nz\" (UniqueName: \"kubernetes.io/projected/62fdeac2-769b-411d-b103-9ca91d804f1f-kube-api-access-5x9nz\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.722679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-var-lib-cni-bin\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.722679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-sys\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.722679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0a651f9-e86d-459b-b079-d9a009db6c7c-agent-certs\") pod \"konnectivity-agent-zdcg9\" (UID: \"a0a651f9-e86d-459b-b079-d9a009db6c7c\") " pod="kube-system/konnectivity-agent-zdcg9"
Sep 11 18:15:25.722679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/608a1286-9f29-437e-b113-0f4cf3015c7b-system-cni-dir\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.722679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9b6c\" (UniqueName: \"kubernetes.io/projected/8de484a7-3da4-4dd7-82da-46fe04ad4508-kube-api-access-x9b6c\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.722679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-sysctl-conf\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.722679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722533    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-var-lib-cni-bin\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.722679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722470    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-sys\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.722679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722543    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 18:15:25.722679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-var-lib-kubelet\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.722679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpwwb\" (UniqueName: \"kubernetes.io/projected/f9719c07-ac09-4ed0-a4da-515e4d30fe2c-kube-api-access-wpwwb\") pod \"node-ca-xvv2h\" (UID: \"f9719c07-ac09-4ed0-a4da-515e4d30fe2c\") " pod="openshift-image-registry/node-ca-xvv2h"
Sep 11 18:15:25.722679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/608a1286-9f29-437e-b113-0f4cf3015c7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.723287 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-slash\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.723287 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-sysctl-conf\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.723287 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.722903    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7a3a8fa6-2a7b-4d7a-ac23-63babb814575-iptables-alerter-script\") pod \"iptables-alerter-q8nvm\" (UID: \"7a3a8fa6-2a7b-4d7a-ac23-63babb814575\") " pod="openshift-network-operator/iptables-alerter-q8nvm"
Sep 11 18:15:25.723450 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-cnibin\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.723450 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723387    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f9719c07-ac09-4ed0-a4da-515e4d30fe2c-serviceca\") pod \"node-ca-xvv2h\" (UID: \"f9719c07-ac09-4ed0-a4da-515e4d30fe2c\") " pod="openshift-image-registry/node-ca-xvv2h"
Sep 11 18:15:25.723450 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-var-lib-kubelet\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.723648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723505    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-slash\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.723648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/608a1286-9f29-437e-b113-0f4cf3015c7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.723648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723532    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-cnibin\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.723648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723570    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-run-ovn\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.723648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723573    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/62fdeac2-769b-411d-b103-9ca91d804f1f-ovnkube-script-lib\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.723648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723620    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-run-ovn\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.723980 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723660    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-multus-socket-dir-parent\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.723980 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-etc-kubernetes\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.723980 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723843    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-etc-kubernetes\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.723980 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723845    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb4f3610-8190-4056-88be-96c5087cb949-dbus\") pod \"global-pull-secret-syncer-9rs25\" (UID: \"eb4f3610-8190-4056-88be-96c5087cb949\") " pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:25.723980 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723846    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-multus-socket-dir-parent\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.723980 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-systemd-units\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.723980 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723936    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-var-lib-openvswitch\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.723980 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62fdeac2-769b-411d-b103-9ca91d804f1f-ovnkube-config\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.723980 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723974    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-var-lib-openvswitch\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.724483 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62fdeac2-769b-411d-b103-9ca91d804f1f-env-overrides\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.724483 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.723973    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-systemd-units\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.724483 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab698762-88b3-4098-bf05-6718e735177a-multus-daemon-config\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.724483 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8p9n\" (UniqueName: \"kubernetes.io/projected/68330d27-5afb-44d1-96a2-774104ee6c4c-kube-api-access-p8p9n\") pod \"node-resolver-b9dfv\" (UID: \"68330d27-5afb-44d1-96a2-774104ee6c4c\") " pod="openshift-dns/node-resolver-b9dfv"
Sep 11 18:15:25.724483 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724167    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-run-systemd\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.724483 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-run-k8s-cni-cncf-io\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.724483 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-socket-dir\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.724483 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-run-netns\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.724483 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-device-dir\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.724483 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68330d27-5afb-44d1-96a2-774104ee6c4c-tmp-dir\") pod \"node-resolver-b9dfv\" (UID: \"68330d27-5afb-44d1-96a2-774104ee6c4c\") " pod="openshift-dns/node-resolver-b9dfv"
Sep 11 18:15:25.724483 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724342    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-lib-modules\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.724483 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb4f3610-8190-4056-88be-96c5087cb949-kubelet-config\") pod \"global-pull-secret-syncer-9rs25\" (UID: \"eb4f3610-8190-4056-88be-96c5087cb949\") " pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:25.724483 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-hostroot\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.724483 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-run-multus-certs\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.724483 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-etc-selinux\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724556    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-tuned\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9719c07-ac09-4ed0-a4da-515e4d30fe2c-host\") pod \"node-ca-xvv2h\" (UID: \"f9719c07-ac09-4ed0-a4da-515e4d30fe2c\") " pod="openshift-image-registry/node-ca-xvv2h"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qgq7\" (UniqueName: \"kubernetes.io/projected/7a3a8fa6-2a7b-4d7a-ac23-63babb814575-kube-api-access-6qgq7\") pod \"iptables-alerter-q8nvm\" (UID: \"7a3a8fa6-2a7b-4d7a-ac23-63babb814575\") " pod="openshift-network-operator/iptables-alerter-q8nvm"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724605    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/62fdeac2-769b-411d-b103-9ca91d804f1f-ovnkube-config\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-modprobe-d\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab698762-88b3-4098-bf05-6718e735177a-cni-binary-copy\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724679    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/62fdeac2-769b-411d-b103-9ca91d804f1f-env-overrides\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724731    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-hostroot\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-lib-modules\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724789    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-run-k8s-cni-cncf-io\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724833    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-run-systemd\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-socket-dir\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724936    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-run-netns\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724982    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-device-dir\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.725356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.724686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj7cz\" (UniqueName: \"kubernetes.io/projected/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-kube-api-access-dj7cz\") pod \"network-metrics-daemon-48sl5\" (UID: \"db03c3a4-cfd5-443f-afe5-6e2fad4b0678\") " pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ab698762-88b3-4098-bf05-6718e735177a-multus-daemon-config\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a3a8fa6-2a7b-4d7a-ac23-63babb814575-host-slash\") pod \"iptables-alerter-q8nvm\" (UID: \"7a3a8fa6-2a7b-4d7a-ac23-63babb814575\") " pod="openshift-network-operator/iptables-alerter-q8nvm"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-etc-selinux\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725101    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7a3a8fa6-2a7b-4d7a-ac23-63babb814575-host-slash\") pod \"iptables-alerter-q8nvm\" (UID: \"7a3a8fa6-2a7b-4d7a-ac23-63babb814575\") " pod="openshift-network-operator/iptables-alerter-q8nvm"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725191    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-run-multus-certs\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-modprobe-d\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-sysctl-d\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68330d27-5afb-44d1-96a2-774104ee6c4c-tmp-dir\") pod \"node-resolver-b9dfv\" (UID: \"68330d27-5afb-44d1-96a2-774104ee6c4c\") " pod="openshift-dns/node-resolver-b9dfv"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d282c\" (UniqueName: \"kubernetes.io/projected/1462a6c5-66bd-4a38-ac43-a80933ec590d-kube-api-access-d282c\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725340    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-sysconfig\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-node-log\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725374    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-sysctl-d\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-cni-netd\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-sysconfig\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-system-cni-dir\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-var-lib-kubelet\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxdjs\" (UniqueName: \"kubernetes.io/projected/ab698762-88b3-4098-bf05-6718e735177a-kube-api-access-wxdjs\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.726269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-system-cni-dir\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.727147 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs\") pod \"network-metrics-daemon-48sl5\" (UID: \"db03c3a4-cfd5-443f-afe5-6e2fad4b0678\") " pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:25.727147 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:25.726329    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:25.727147 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.726386    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-node-log\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.727147 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.725603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-sys-fs\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.727147 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:25.726433    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs podName:db03c3a4-cfd5-443f-afe5-6e2fad4b0678 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:26.226399378 +0000 UTC m=+2.967526222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs") pod "network-metrics-daemon-48sl5" (UID: "db03c3a4-cfd5-443f-afe5-6e2fad4b0678") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:25.727147 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.726553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ab698762-88b3-4098-bf05-6718e735177a-cni-binary-copy\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.727147 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.726787    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-cni-netd\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.727527 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.727493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1462a6c5-66bd-4a38-ac43-a80933ec590d-tmp\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.727605 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.726429    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-var-lib-kubelet\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.727687 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.727668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-kubelet\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.727773 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.727701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-run-openvswitch\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.727773 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.727730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-registration-dir\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.727773 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.727764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-systemd\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.727916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.727794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-run\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.727916 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.727861    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-run-openvswitch\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.727993 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.727916    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-kubelet\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.727993 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.727929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-tuned\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.727993 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.727939    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-run\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.727993 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.727961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/608a1286-9f29-437e-b113-0f4cf3015c7b-cnibin\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.727993 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.727989    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-registration-dir\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.728235 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728154    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-systemd\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.728235 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728200    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/608a1286-9f29-437e-b113-0f4cf3015c7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.728362 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62fdeac2-769b-411d-b103-9ca91d804f1f-ovn-node-metrics-cert\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.728362 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grb7r\" (UniqueName: \"kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r\") pod \"network-check-target-fxtlq\" (UID: \"a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b\") " pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:25.728362 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728315    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0a651f9-e86d-459b-b079-d9a009db6c7c-konnectivity-ca\") pod \"konnectivity-agent-zdcg9\" (UID: \"a0a651f9-e86d-459b-b079-d9a009db6c7c\") " pod="kube-system/konnectivity-agent-zdcg9"
Sep 11 18:15:25.728362 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-etc-openvswitch\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.728695 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-multus-cni-dir\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.728695 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728416    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-multus-conf-dir\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.728695 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/68330d27-5afb-44d1-96a2-774104ee6c4c-hosts-file\") pod \"node-resolver-b9dfv\" (UID: \"68330d27-5afb-44d1-96a2-774104ee6c4c\") " pod="openshift-dns/node-resolver-b9dfv"
Sep 11 18:15:25.728695 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-host\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.728695 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret\") pod \"global-pull-secret-syncer-9rs25\" (UID: \"eb4f3610-8190-4056-88be-96c5087cb949\") " pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:25.728695 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.728695 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-kubernetes\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.728695 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwbmh\" (UniqueName: \"kubernetes.io/projected/608a1286-9f29-437e-b113-0f4cf3015c7b-kube-api-access-kwbmh\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.729118 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-run-netns\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.729118 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-var-lib-cni-multus\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.729118 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728791    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/68330d27-5afb-44d1-96a2-774104ee6c4c-hosts-file\") pod \"node-resolver-b9dfv\" (UID: \"68330d27-5afb-44d1-96a2-774104ee6c4c\") " pod="openshift-dns/node-resolver-b9dfv"
Sep 11 18:15:25.729118 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728904    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-var-lib-cni-multus\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.729118 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.728978    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-host\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.729118 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.729054    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.729471 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.729120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1462a6c5-66bd-4a38-ac43-a80933ec590d-etc-kubernetes\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.729471 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.729163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/62fdeac2-769b-411d-b103-9ca91d804f1f-etc-openvswitch\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.729471 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.729253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-host-run-netns\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.729471 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.729330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-multus-cni-dir\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.729471 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.729394    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ab698762-88b3-4098-bf05-6718e735177a-multus-conf-dir\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.729766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.729516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8de484a7-3da4-4dd7-82da-46fe04ad4508-sys-fs\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.733427 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.733400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x9nz\" (UniqueName: \"kubernetes.io/projected/62fdeac2-769b-411d-b103-9ca91d804f1f-kube-api-access-5x9nz\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.734988 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:25.734966    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:25.734988 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:25.734995    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:25.734988 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:25.735009    2581 projected.go:196] Error preparing data for projected volume kube-api-access-grb7r for pod openshift-network-diagnostics/network-check-target-fxtlq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:25.735841 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:25.735817    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r podName:a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b nodeName:}" failed. No retries permitted until 2026-09-11 18:15:26.235793462 +0000 UTC m=+2.976920308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-grb7r" (UniqueName: "kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r") pod "network-check-target-fxtlq" (UID: "a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:25.736913 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.736881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8p9n\" (UniqueName: \"kubernetes.io/projected/68330d27-5afb-44d1-96a2-774104ee6c4c-kube-api-access-p8p9n\") pod \"node-resolver-b9dfv\" (UID: \"68330d27-5afb-44d1-96a2-774104ee6c4c\") " pod="openshift-dns/node-resolver-b9dfv"
Sep 11 18:15:25.737026 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.736924    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d282c\" (UniqueName: \"kubernetes.io/projected/1462a6c5-66bd-4a38-ac43-a80933ec590d-kube-api-access-d282c\") pod \"tuned-lw4pm\" (UID: \"1462a6c5-66bd-4a38-ac43-a80933ec590d\") " pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.738202 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.738182    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9b6c\" (UniqueName: \"kubernetes.io/projected/8de484a7-3da4-4dd7-82da-46fe04ad4508-kube-api-access-x9b6c\") pod \"aws-ebs-csi-driver-node-8xmwg\" (UID: \"8de484a7-3da4-4dd7-82da-46fe04ad4508\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.738316 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.738263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/62fdeac2-769b-411d-b103-9ca91d804f1f-ovn-node-metrics-cert\") pod \"ovnkube-node-mx4jb\" (UID: \"62fdeac2-769b-411d-b103-9ca91d804f1f\") " pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.738602 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.738584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qgq7\" (UniqueName: \"kubernetes.io/projected/7a3a8fa6-2a7b-4d7a-ac23-63babb814575-kube-api-access-6qgq7\") pod \"iptables-alerter-q8nvm\" (UID: \"7a3a8fa6-2a7b-4d7a-ac23-63babb814575\") " pod="openshift-network-operator/iptables-alerter-q8nvm"
Sep 11 18:15:25.739311 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.739290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxdjs\" (UniqueName: \"kubernetes.io/projected/ab698762-88b3-4098-bf05-6718e735177a-kube-api-access-wxdjs\") pod \"multus-sz67x\" (UID: \"ab698762-88b3-4098-bf05-6718e735177a\") " pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.740539 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.740516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj7cz\" (UniqueName: \"kubernetes.io/projected/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-kube-api-access-dj7cz\") pod \"network-metrics-daemon-48sl5\" (UID: \"db03c3a4-cfd5-443f-afe5-6e2fad4b0678\") " pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:25.761835 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.761786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal" event={"ID":"e76cbbfe0210854c66f75b5edb339223","Type":"ContainerStarted","Data":"6b1dba8dd6d0d8aaea01d9888862fe306075356a1da63a2eaebfb5d312657810"}
Sep 11 18:15:25.762909 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.762854    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:10:25 +0000 UTC" deadline="2028-04-24 20:26:36.746173819 +0000 UTC"
Sep 11 18:15:25.762909 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.762908    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14186h11m10.983270927s"
Sep 11 18:15:25.763338 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.763311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-166.ec2.internal" event={"ID":"1fccd7d8530b9bbb746ecca8d0ec2393","Type":"ContainerStarted","Data":"b247c904ac13a9091e9927cb3f3428fd54c20a85a9713b4fe7a6cbd4bb41da56"}
Sep 11 18:15:25.829758 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.829728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb4f3610-8190-4056-88be-96c5087cb949-kubelet-config\") pod \"global-pull-secret-syncer-9rs25\" (UID: \"eb4f3610-8190-4056-88be-96c5087cb949\") " pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:25.829908 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.829766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9719c07-ac09-4ed0-a4da-515e4d30fe2c-host\") pod \"node-ca-xvv2h\" (UID: \"f9719c07-ac09-4ed0-a4da-515e4d30fe2c\") " pod="openshift-image-registry/node-ca-xvv2h"
Sep 11 18:15:25.829908 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.829821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/608a1286-9f29-437e-b113-0f4cf3015c7b-cnibin\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.829908 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.829841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eb4f3610-8190-4056-88be-96c5087cb949-kubelet-config\") pod \"global-pull-secret-syncer-9rs25\" (UID: \"eb4f3610-8190-4056-88be-96c5087cb949\") " pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:25.829908 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.829849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/608a1286-9f29-437e-b113-0f4cf3015c7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.829908 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.829878    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f9719c07-ac09-4ed0-a4da-515e4d30fe2c-host\") pod \"node-ca-xvv2h\" (UID: \"f9719c07-ac09-4ed0-a4da-515e4d30fe2c\") " pod="openshift-image-registry/node-ca-xvv2h"
Sep 11 18:15:25.830120 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.829902    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/608a1286-9f29-437e-b113-0f4cf3015c7b-cnibin\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.830120 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.829924    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0a651f9-e86d-459b-b079-d9a009db6c7c-konnectivity-ca\") pod \"konnectivity-agent-zdcg9\" (UID: \"a0a651f9-e86d-459b-b079-d9a009db6c7c\") " pod="kube-system/konnectivity-agent-zdcg9"
Sep 11 18:15:25.830120 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.829959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret\") pod \"global-pull-secret-syncer-9rs25\" (UID: \"eb4f3610-8190-4056-88be-96c5087cb949\") " pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:25.830120 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.830002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwbmh\" (UniqueName: \"kubernetes.io/projected/608a1286-9f29-437e-b113-0f4cf3015c7b-kube-api-access-kwbmh\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.830120 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.830033    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/608a1286-9f29-437e-b113-0f4cf3015c7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.830120 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.830064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0a651f9-e86d-459b-b079-d9a009db6c7c-agent-certs\") pod \"konnectivity-agent-zdcg9\" (UID: \"a0a651f9-e86d-459b-b079-d9a009db6c7c\") " pod="kube-system/konnectivity-agent-zdcg9"
Sep 11 18:15:25.830120 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.830088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/608a1286-9f29-437e-b113-0f4cf3015c7b-system-cni-dir\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.830372 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.830124    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpwwb\" (UniqueName: \"kubernetes.io/projected/f9719c07-ac09-4ed0-a4da-515e4d30fe2c-kube-api-access-wpwwb\") pod \"node-ca-xvv2h\" (UID: \"f9719c07-ac09-4ed0-a4da-515e4d30fe2c\") " pod="openshift-image-registry/node-ca-xvv2h"
Sep 11 18:15:25.830372 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.830152    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/608a1286-9f29-437e-b113-0f4cf3015c7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.830372 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.830179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f9719c07-ac09-4ed0-a4da-515e4d30fe2c-serviceca\") pod \"node-ca-xvv2h\" (UID: \"f9719c07-ac09-4ed0-a4da-515e4d30fe2c\") " pod="openshift-image-registry/node-ca-xvv2h"
Sep 11 18:15:25.830372 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.830204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/608a1286-9f29-437e-b113-0f4cf3015c7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.830372 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.830232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb4f3610-8190-4056-88be-96c5087cb949-dbus\") pod \"global-pull-secret-syncer-9rs25\" (UID: \"eb4f3610-8190-4056-88be-96c5087cb949\") " pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:25.830372 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.830336    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eb4f3610-8190-4056-88be-96c5087cb949-dbus\") pod \"global-pull-secret-syncer-9rs25\" (UID: \"eb4f3610-8190-4056-88be-96c5087cb949\") " pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:25.830624 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.830401    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/608a1286-9f29-437e-b113-0f4cf3015c7b-system-cni-dir\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.830624 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.830405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/608a1286-9f29-437e-b113-0f4cf3015c7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.830624 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.830477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a0a651f9-e86d-459b-b079-d9a009db6c7c-konnectivity-ca\") pod \"konnectivity-agent-zdcg9\" (UID: \"a0a651f9-e86d-459b-b079-d9a009db6c7c\") " pod="kube-system/konnectivity-agent-zdcg9"
Sep 11 18:15:25.830624 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:25.830532    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:25.830624 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:25.830584    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret podName:eb4f3610-8190-4056-88be-96c5087cb949 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:26.330567787 +0000 UTC m=+3.071694616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret") pod "global-pull-secret-syncer-9rs25" (UID: "eb4f3610-8190-4056-88be-96c5087cb949") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:25.830843 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.830655    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/608a1286-9f29-437e-b113-0f4cf3015c7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.830843 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.830749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f9719c07-ac09-4ed0-a4da-515e4d30fe2c-serviceca\") pod \"node-ca-xvv2h\" (UID: \"f9719c07-ac09-4ed0-a4da-515e4d30fe2c\") " pod="openshift-image-registry/node-ca-xvv2h"
Sep 11 18:15:25.831180 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.831148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/608a1286-9f29-437e-b113-0f4cf3015c7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.831296 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.831256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/608a1286-9f29-437e-b113-0f4cf3015c7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.833701 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.833681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a0a651f9-e86d-459b-b079-d9a009db6c7c-agent-certs\") pod \"konnectivity-agent-zdcg9\" (UID: \"a0a651f9-e86d-459b-b079-d9a009db6c7c\") " pod="kube-system/konnectivity-agent-zdcg9"
Sep 11 18:15:25.846138 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.846103    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwbmh\" (UniqueName: \"kubernetes.io/projected/608a1286-9f29-437e-b113-0f4cf3015c7b-kube-api-access-kwbmh\") pod \"multus-additional-cni-plugins-p6fm6\" (UID: \"608a1286-9f29-437e-b113-0f4cf3015c7b\") " pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.847002 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.846980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpwwb\" (UniqueName: \"kubernetes.io/projected/f9719c07-ac09-4ed0-a4da-515e4d30fe2c-kube-api-access-wpwwb\") pod \"node-ca-xvv2h\" (UID: \"f9719c07-ac09-4ed0-a4da-515e4d30fe2c\") " pod="openshift-image-registry/node-ca-xvv2h"
Sep 11 18:15:25.909224 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.909192    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:25.916352 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.916330    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sz67x"
Sep 11 18:15:25.927978 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.927957    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg"
Sep 11 18:15:25.934825 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.934805    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b9dfv"
Sep 11 18:15:25.945571 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.945552    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q8nvm"
Sep 11 18:15:25.955345 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.955327    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lw4pm"
Sep 11 18:15:25.965913 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.965894    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xvv2h"
Sep 11 18:15:25.973667 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.973641    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zdcg9"
Sep 11 18:15:25.980432 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.980416    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-p6fm6"
Sep 11 18:15:25.997656 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:25.997630    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:26.233084 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:26.233015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs\") pod \"network-metrics-daemon-48sl5\" (UID: \"db03c3a4-cfd5-443f-afe5-6e2fad4b0678\") " pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:26.233221 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:26.233197    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:26.233272 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:26.233261    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs podName:db03c3a4-cfd5-443f-afe5-6e2fad4b0678 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:27.233243806 +0000 UTC m=+3.974370628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs") pod "network-metrics-daemon-48sl5" (UID: "db03c3a4-cfd5-443f-afe5-6e2fad4b0678") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:26.333900 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:26.333864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grb7r\" (UniqueName: \"kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r\") pod \"network-check-target-fxtlq\" (UID: \"a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b\") " pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:26.334066 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:26.333920    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret\") pod \"global-pull-secret-syncer-9rs25\" (UID: \"eb4f3610-8190-4056-88be-96c5087cb949\") " pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:26.334066 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:26.334040    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:26.334066 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:26.334046    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:26.334066 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:26.334067    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:26.334226 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:26.334078    2581 projected.go:196] Error preparing data for projected volume kube-api-access-grb7r for pod openshift-network-diagnostics/network-check-target-fxtlq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:26.334226 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:26.334114    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret podName:eb4f3610-8190-4056-88be-96c5087cb949 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:27.334093616 +0000 UTC m=+4.075220459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret") pod "global-pull-secret-syncer-9rs25" (UID: "eb4f3610-8190-4056-88be-96c5087cb949") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:26.334226 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:26.334137    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r podName:a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b nodeName:}" failed. No retries permitted until 2026-09-11 18:15:27.334126075 +0000 UTC m=+4.075252898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-grb7r" (UniqueName: "kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r") pod "network-check-target-fxtlq" (UID: "a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:26.758129 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:26.758097    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:26.758672 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:26.758221    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:26.763271 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:26.763235    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:10:25 +0000 UTC" deadline="2028-05-27 03:58:28.728679222 +0000 UTC"
Sep 11 18:15:26.763401 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:26.763272    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14961h43m1.965412856s"
Sep 11 18:15:26.828759 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:26.828719    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68330d27_5afb_44d1_96a2_774104ee6c4c.slice/crio-7d722174d43844ef703d6071437514f28ad56e143b46581858e145b375f688ca WatchSource:0}: Error finding container 7d722174d43844ef703d6071437514f28ad56e143b46581858e145b375f688ca: Status 404 returned error can't find the container with id 7d722174d43844ef703d6071437514f28ad56e143b46581858e145b375f688ca
Sep 11 18:15:26.837059 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:26.837028    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fdeac2_769b_411d_b103_9ca91d804f1f.slice/crio-6e80584a54bb1e9d5cb3263c128855ccb3543a394809eca896bb2c8bcb2d38dc WatchSource:0}: Error finding container 6e80584a54bb1e9d5cb3263c128855ccb3543a394809eca896bb2c8bcb2d38dc: Status 404 returned error can't find the container with id 6e80584a54bb1e9d5cb3263c128855ccb3543a394809eca896bb2c8bcb2d38dc
Sep 11 18:15:26.838253 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:26.838227    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3a8fa6_2a7b_4d7a_ac23_63babb814575.slice/crio-5e988b08aaa0472c4390ed08a6ee2c1d176fc898e469bb152a56b3d9ff03f6fd WatchSource:0}: Error finding container 5e988b08aaa0472c4390ed08a6ee2c1d176fc898e469bb152a56b3d9ff03f6fd: Status 404 returned error can't find the container with id 5e988b08aaa0472c4390ed08a6ee2c1d176fc898e469bb152a56b3d9ff03f6fd
Sep 11 18:15:26.839371 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:26.839291    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a651f9_e86d_459b_b079_d9a009db6c7c.slice/crio-de2bf373d0755c05c0428ce93e327079d69d8f569f31ced1bde0ade881a0ccf6 WatchSource:0}: Error finding container de2bf373d0755c05c0428ce93e327079d69d8f569f31ced1bde0ade881a0ccf6: Status 404 returned error can't find the container with id de2bf373d0755c05c0428ce93e327079d69d8f569f31ced1bde0ade881a0ccf6
Sep 11 18:15:26.840093 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:26.839927    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9719c07_ac09_4ed0_a4da_515e4d30fe2c.slice/crio-125b67ae55685a4c02fd91b1c59f581840c63856c9efe5cc9e7e979cc6a7c057 WatchSource:0}: Error finding container 125b67ae55685a4c02fd91b1c59f581840c63856c9efe5cc9e7e979cc6a7c057: Status 404 returned error can't find the container with id 125b67ae55685a4c02fd91b1c59f581840c63856c9efe5cc9e7e979cc6a7c057
Sep 11 18:15:26.841633 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:26.841509    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab698762_88b3_4098_bf05_6718e735177a.slice/crio-295f688875c85ecca60c78301643f9e014d9d1bda854073ba19c5c634e2d7222 WatchSource:0}: Error finding container 295f688875c85ecca60c78301643f9e014d9d1bda854073ba19c5c634e2d7222: Status 404 returned error can't find the container with id 295f688875c85ecca60c78301643f9e014d9d1bda854073ba19c5c634e2d7222
Sep 11 18:15:26.842805 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:26.842777    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608a1286_9f29_437e_b113_0f4cf3015c7b.slice/crio-2233f32a8e4851955c30a45a367ac45977810b7956bc1978b156c38b38374d72 WatchSource:0}: Error finding container 2233f32a8e4851955c30a45a367ac45977810b7956bc1978b156c38b38374d72: Status 404 returned error can't find the container with id 2233f32a8e4851955c30a45a367ac45977810b7956bc1978b156c38b38374d72
Sep 11 18:15:27.241901 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.241716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs\") pod \"network-metrics-daemon-48sl5\" (UID: \"db03c3a4-cfd5-443f-afe5-6e2fad4b0678\") " pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:27.242029 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:27.241865    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:27.242069 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:27.242045    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs podName:db03c3a4-cfd5-443f-afe5-6e2fad4b0678 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:29.242018336 +0000 UTC m=+5.983145160 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs") pod "network-metrics-daemon-48sl5" (UID: "db03c3a4-cfd5-443f-afe5-6e2fad4b0678") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:27.342891 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.342851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grb7r\" (UniqueName: \"kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r\") pod \"network-check-target-fxtlq\" (UID: \"a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b\") " pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:27.343053 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.342909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret\") pod \"global-pull-secret-syncer-9rs25\" (UID: \"eb4f3610-8190-4056-88be-96c5087cb949\") " pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:27.343135 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:27.343054    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:27.343135 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:27.343118    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret podName:eb4f3610-8190-4056-88be-96c5087cb949 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:29.34309868 +0000 UTC m=+6.084225510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret") pod "global-pull-secret-syncer-9rs25" (UID: "eb4f3610-8190-4056-88be-96c5087cb949") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:27.343599 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:27.343570    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:27.343705 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:27.343604    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:27.343705 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:27.343618    2581 projected.go:196] Error preparing data for projected volume kube-api-access-grb7r for pod openshift-network-diagnostics/network-check-target-fxtlq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:27.343705 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:27.343667    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r podName:a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b nodeName:}" failed. No retries permitted until 2026-09-11 18:15:29.343649371 +0000 UTC m=+6.084776201 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-grb7r" (UniqueName: "kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r") pod "network-check-target-fxtlq" (UID: "a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:27.759925 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.759841    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:27.760508 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:27.759974    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:27.760508 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.760378    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:27.760508 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:27.760480    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:27.787876 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.787843    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xvv2h" event={"ID":"f9719c07-ac09-4ed0-a4da-515e4d30fe2c","Type":"ContainerStarted","Data":"125b67ae55685a4c02fd91b1c59f581840c63856c9efe5cc9e7e979cc6a7c057"}
Sep 11 18:15:27.794379 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.794349    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lw4pm" event={"ID":"1462a6c5-66bd-4a38-ac43-a80933ec590d","Type":"ContainerStarted","Data":"ce8c270b9ab78d45dc8b1d5d07527012c5eaa313c7b97c8f4b857dbc7ca2d647"}
Sep 11 18:15:27.822675 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.822647    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b9dfv" event={"ID":"68330d27-5afb-44d1-96a2-774104ee6c4c","Type":"ContainerStarted","Data":"7d722174d43844ef703d6071437514f28ad56e143b46581858e145b375f688ca"}
Sep 11 18:15:27.826579 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.826246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg" event={"ID":"8de484a7-3da4-4dd7-82da-46fe04ad4508","Type":"ContainerStarted","Data":"1bc9e9206b35a550ac2d6f790ad61b06bdbd237eb62c62f9ab3dbd50c85aad71"}
Sep 11 18:15:27.830436 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.830191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-166.ec2.internal" event={"ID":"1fccd7d8530b9bbb746ecca8d0ec2393","Type":"ContainerStarted","Data":"6a65cff6937e66fe5de3856cd26b28879833c310a3622bd483099214b9959ebe"}
Sep 11 18:15:27.833200 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.833160    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6fm6" event={"ID":"608a1286-9f29-437e-b113-0f4cf3015c7b","Type":"ContainerStarted","Data":"2233f32a8e4851955c30a45a367ac45977810b7956bc1978b156c38b38374d72"}
Sep 11 18:15:27.836485 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.835341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sz67x" event={"ID":"ab698762-88b3-4098-bf05-6718e735177a","Type":"ContainerStarted","Data":"295f688875c85ecca60c78301643f9e014d9d1bda854073ba19c5c634e2d7222"}
Sep 11 18:15:27.838372 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.838346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zdcg9" event={"ID":"a0a651f9-e86d-459b-b079-d9a009db6c7c","Type":"ContainerStarted","Data":"de2bf373d0755c05c0428ce93e327079d69d8f569f31ced1bde0ade881a0ccf6"}
Sep 11 18:15:27.841100 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.841049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q8nvm" event={"ID":"7a3a8fa6-2a7b-4d7a-ac23-63babb814575","Type":"ContainerStarted","Data":"5e988b08aaa0472c4390ed08a6ee2c1d176fc898e469bb152a56b3d9ff03f6fd"}
Sep 11 18:15:27.847510 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.847401    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb" event={"ID":"62fdeac2-769b-411d-b103-9ca91d804f1f","Type":"ContainerStarted","Data":"6e80584a54bb1e9d5cb3263c128855ccb3543a394809eca896bb2c8bcb2d38dc"}
Sep 11 18:15:27.866617 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.865904    2581 generic.go:356] "Generic (PLEG): container finished" podID="e76cbbfe0210854c66f75b5edb339223" containerID="49253e3c3278568237ffc31aef91fd0239b27583bdc555bc5df3b7b17ed8e15c" exitCode=0
Sep 11 18:15:27.866617 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.865941    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal" event={"ID":"e76cbbfe0210854c66f75b5edb339223","Type":"ContainerDied","Data":"49253e3c3278568237ffc31aef91fd0239b27583bdc555bc5df3b7b17ed8e15c"}
Sep 11 18:15:27.884589 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:27.883788    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-166.ec2.internal" podStartSLOduration=2.883768743 podStartE2EDuration="2.883768743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:27.851818751 +0000 UTC m=+4.592945600" watchObservedRunningTime="2026-09-11 18:15:27.883768743 +0000 UTC m=+4.624895593"
Sep 11 18:15:28.758606 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:28.757812    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:28.758606 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:28.757954    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:28.898124 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:28.898091    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal" event={"ID":"e76cbbfe0210854c66f75b5edb339223","Type":"ContainerStarted","Data":"3be34ff147d3b94f84f1b28ec38a44e75395ae282d44a973783da377201d011d"}
Sep 11 18:15:28.916271 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:28.916214    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-166.ec2.internal" podStartSLOduration=3.916195929 podStartE2EDuration="3.916195929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:28.915364881 +0000 UTC m=+5.656491727" watchObservedRunningTime="2026-09-11 18:15:28.916195929 +0000 UTC m=+5.657322801"
Sep 11 18:15:29.263409 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:29.262380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs\") pod \"network-metrics-daemon-48sl5\" (UID: \"db03c3a4-cfd5-443f-afe5-6e2fad4b0678\") " pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:29.263409 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:29.262592    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:29.263409 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:29.262657    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs podName:db03c3a4-cfd5-443f-afe5-6e2fad4b0678 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:33.262637266 +0000 UTC m=+10.003764095 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs") pod "network-metrics-daemon-48sl5" (UID: "db03c3a4-cfd5-443f-afe5-6e2fad4b0678") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:29.363286 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:29.363244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grb7r\" (UniqueName: \"kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r\") pod \"network-check-target-fxtlq\" (UID: \"a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b\") " pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:29.363477 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:29.363302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret\") pod \"global-pull-secret-syncer-9rs25\" (UID: \"eb4f3610-8190-4056-88be-96c5087cb949\") " pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:29.363477 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:29.363452    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:29.363622 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:29.363528    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret podName:eb4f3610-8190-4056-88be-96c5087cb949 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:33.363498436 +0000 UTC m=+10.104625263 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret") pod "global-pull-secret-syncer-9rs25" (UID: "eb4f3610-8190-4056-88be-96c5087cb949") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:29.364203 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:29.364075    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:29.364203 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:29.364104    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:29.364203 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:29.364118    2581 projected.go:196] Error preparing data for projected volume kube-api-access-grb7r for pod openshift-network-diagnostics/network-check-target-fxtlq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:29.364203 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:29.364169    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r podName:a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b nodeName:}" failed. No retries permitted until 2026-09-11 18:15:33.364152314 +0000 UTC m=+10.105279143 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-grb7r" (UniqueName: "kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r") pod "network-check-target-fxtlq" (UID: "a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:29.760775 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:29.760689    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:29.760949 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:29.760831    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:29.761233 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:29.761207    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:29.761338 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:29.761310    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:30.758505 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:30.757961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:30.758505 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:30.758105    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:31.758404 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:31.758371    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:31.758607 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:31.758525    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:31.759181 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:31.759019    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:31.759181 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:31.759138    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:32.758359 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:32.757840    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:32.758359 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:32.757999    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:33.296890 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:33.296854    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs\") pod \"network-metrics-daemon-48sl5\" (UID: \"db03c3a4-cfd5-443f-afe5-6e2fad4b0678\") " pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:33.297368 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:33.296976    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:33.297368 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:33.297048    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs podName:db03c3a4-cfd5-443f-afe5-6e2fad4b0678 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:41.297025332 +0000 UTC m=+18.038152166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs") pod "network-metrics-daemon-48sl5" (UID: "db03c3a4-cfd5-443f-afe5-6e2fad4b0678") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:33.398055 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:33.398014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grb7r\" (UniqueName: \"kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r\") pod \"network-check-target-fxtlq\" (UID: \"a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b\") " pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:33.398232 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:33.398066    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret\") pod \"global-pull-secret-syncer-9rs25\" (UID: \"eb4f3610-8190-4056-88be-96c5087cb949\") " pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:33.398232 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:33.398217    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:33.398367 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:33.398243    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:33.398367 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:33.398258    2581 projected.go:196] Error preparing data for projected volume kube-api-access-grb7r for pod openshift-network-diagnostics/network-check-target-fxtlq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:33.398367 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:33.398217    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:33.398367 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:33.398309    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r podName:a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b nodeName:}" failed. No retries permitted until 2026-09-11 18:15:41.398290453 +0000 UTC m=+18.139417500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-grb7r" (UniqueName: "kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r") pod "network-check-target-fxtlq" (UID: "a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:33.398367 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:33.398336    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret podName:eb4f3610-8190-4056-88be-96c5087cb949 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:41.398324243 +0000 UTC m=+18.139451066 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret") pod "global-pull-secret-syncer-9rs25" (UID: "eb4f3610-8190-4056-88be-96c5087cb949") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:33.759468 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:33.759427    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:33.759659 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:33.759550    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:33.759659 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:33.759612    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:33.759755 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:33.759699    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:34.758636 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:34.758184    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:34.758636 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:34.758331    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:35.759729 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:35.759686    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:35.760196 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:35.759832    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:35.760196 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:35.759883    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:35.760196 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:35.760000    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:36.758330 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:36.758299    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:36.758468 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:36.758405    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:37.757976 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:37.757930    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:37.758506 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:37.758051    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:37.758506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:37.758107    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:37.758506 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:37.758199    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:38.757486 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:38.757456    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:38.757614 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:38.757590    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:39.757839 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:39.757731    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:39.758268 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:39.757855    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:39.758268 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:39.757909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:39.758268 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:39.758028    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:40.758131 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:40.758102    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:40.758640 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:40.758219    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:41.358427 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:41.358388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs\") pod \"network-metrics-daemon-48sl5\" (UID: \"db03c3a4-cfd5-443f-afe5-6e2fad4b0678\") " pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:41.358674 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:41.358560    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:41.358674 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:41.358627    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs podName:db03c3a4-cfd5-443f-afe5-6e2fad4b0678 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:57.358610001 +0000 UTC m=+34.099736823 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs") pod "network-metrics-daemon-48sl5" (UID: "db03c3a4-cfd5-443f-afe5-6e2fad4b0678") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:41.459042 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:41.459014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grb7r\" (UniqueName: \"kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r\") pod \"network-check-target-fxtlq\" (UID: \"a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b\") " pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:41.459042 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:41.459052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret\") pod \"global-pull-secret-syncer-9rs25\" (UID: \"eb4f3610-8190-4056-88be-96c5087cb949\") " pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:41.459273 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:41.459181    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:41.459273 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:41.459228    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:41.459273 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:41.459240    2581 projected.go:196] Error preparing data for projected volume kube-api-access-grb7r for pod openshift-network-diagnostics/network-check-target-fxtlq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:41.459412 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:41.459286    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r podName:a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b nodeName:}" failed. No retries permitted until 2026-09-11 18:15:57.459269863 +0000 UTC m=+34.200396685 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-grb7r" (UniqueName: "kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r") pod "network-check-target-fxtlq" (UID: "a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:41.459412 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:41.459191    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:41.459412 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:41.459354    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret podName:eb4f3610-8190-4056-88be-96c5087cb949 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:57.459338313 +0000 UTC m=+34.200465154 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret") pod "global-pull-secret-syncer-9rs25" (UID: "eb4f3610-8190-4056-88be-96c5087cb949") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:41.778004 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:41.777972    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:41.778430 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:41.777972    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:41.778430 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:41.778106    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:41.778430 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:41.778179    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:42.757990 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:42.757957    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:42.758186 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:42.758100    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:43.759075 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:43.759041    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:43.759558 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:43.759141    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:43.759558 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:43.759195    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:43.759558 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:43.759324    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:44.757535 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:44.757496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:44.757722 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:44.757632    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:45.758504 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:45.757763    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:45.758504 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:45.757876    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:45.758504 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:45.758001    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:45.758504 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:45.758084    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:45.947467 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:45.946925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6fm6" event={"ID":"608a1286-9f29-437e-b113-0f4cf3015c7b","Type":"ContainerStarted","Data":"9c24134c8611c76f1502439b00533dc9733f77bce1d4efe3014c550c7105f0ab"}
Sep 11 18:15:45.949580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:45.949501    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sz67x" event={"ID":"ab698762-88b3-4098-bf05-6718e735177a","Type":"ContainerStarted","Data":"9705f41b34c4eb0524a484eeb0cc159c13b460dbfcbdef7adf70de1cf1300d8b"}
Sep 11 18:15:45.951500 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:45.951451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zdcg9" event={"ID":"a0a651f9-e86d-459b-b079-d9a009db6c7c","Type":"ContainerStarted","Data":"872bbf645478c9f8df5332960c566b50e1771f892b83e4108dc22af7042b8e11"}
Sep 11 18:15:45.954805 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:45.954700    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb" event={"ID":"62fdeac2-769b-411d-b103-9ca91d804f1f","Type":"ContainerStarted","Data":"fd49f109b2b2cf9ef8036c06bb5eac52518c7d8b4d1ca9d3cc13dcef3c6ee992"}
Sep 11 18:15:45.957127 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:45.957105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xvv2h" event={"ID":"f9719c07-ac09-4ed0-a4da-515e4d30fe2c","Type":"ContainerStarted","Data":"1f73bbc864bad66ddbe44189b2dde022f2cd3358b01b2b81c379c35d382d31c9"}
Sep 11 18:15:45.959025 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:45.959007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lw4pm" event={"ID":"1462a6c5-66bd-4a38-ac43-a80933ec590d","Type":"ContainerStarted","Data":"bf03a17c82c4c787181a07ca344de60ef51edac83510093d2eb943863d475f02"}
Sep 11 18:15:45.960978 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:45.960954    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b9dfv" event={"ID":"68330d27-5afb-44d1-96a2-774104ee6c4c","Type":"ContainerStarted","Data":"c57b200817317be817a07d08201b4ec0792d610724750a12574e55bcd6e98494"}
Sep 11 18:15:46.009828 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.009353    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zdcg9" podStartSLOduration=3.123090193 podStartE2EDuration="22.009336637s" totalImagesPullingTime="18.886246444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:46.008743311 +0000 UTC m=+22.749870154" watchObservedRunningTime="2026-09-11 18:15:46.009336637 +0000 UTC m=+22.750463480"
Sep 11 18:15:46.009828 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.009635    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xvv2h" podStartSLOduration=3.116278917 podStartE2EDuration="22.009627149s" totalImagesPullingTime="18.893348232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:45.990314961 +0000 UTC m=+22.731441795" watchObservedRunningTime="2026-09-11 18:15:46.009627149 +0000 UTC m=+22.750753992"
Sep 11 18:15:46.024600 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.024561    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-b9dfv" podStartSLOduration=3.128653362 podStartE2EDuration="22.024545791s" totalImagesPullingTime="18.895892429s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:46.024406298 +0000 UTC m=+22.765533138" watchObservedRunningTime="2026-09-11 18:15:46.024545791 +0000 UTC m=+22.765672631"
Sep 11 18:15:46.043230 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.043190    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lw4pm" podStartSLOduration=3.146079635 podStartE2EDuration="22.043174417s" totalImagesPullingTime="18.897094782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:46.043155274 +0000 UTC m=+22.784282117" watchObservedRunningTime="2026-09-11 18:15:46.043174417 +0000 UTC m=+22.784301260"
Sep 11 18:15:46.060146 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.060105    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sz67x" podStartSLOduration=3.117545299 podStartE2EDuration="22.060090257s" totalImagesPullingTime="18.942544958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:46.059643285 +0000 UTC m=+22.800770130" watchObservedRunningTime="2026-09-11 18:15:46.060090257 +0000 UTC m=+22.801217100"
Sep 11 18:15:46.555284 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.555210    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zdcg9"
Sep 11 18:15:46.555870 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.555843    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zdcg9"
Sep 11 18:15:46.757520 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.757494    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:46.757676 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:46.757594    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:46.964303 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.964271    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg" event={"ID":"8de484a7-3da4-4dd7-82da-46fe04ad4508","Type":"ContainerStarted","Data":"e1282bc5849ce205645842220c5ee24428ef75a11fb920047cbe9f843a8a70e2"}
Sep 11 18:15:46.966793 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.966758    2581 generic.go:356] "Generic (PLEG): container finished" podID="608a1286-9f29-437e-b113-0f4cf3015c7b" containerID="9c24134c8611c76f1502439b00533dc9733f77bce1d4efe3014c550c7105f0ab" exitCode=0
Sep 11 18:15:46.966936 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.966874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6fm6" event={"ID":"608a1286-9f29-437e-b113-0f4cf3015c7b","Type":"ContainerDied","Data":"9c24134c8611c76f1502439b00533dc9733f77bce1d4efe3014c550c7105f0ab"}
Sep 11 18:15:46.971344 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.971315    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb" event={"ID":"62fdeac2-769b-411d-b103-9ca91d804f1f","Type":"ContainerStarted","Data":"fb1a0fd03912ea771834b5529dbeefba5cd506280fc47744deb327683be4da71"}
Sep 11 18:15:46.971471 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.971396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb" event={"ID":"62fdeac2-769b-411d-b103-9ca91d804f1f","Type":"ContainerStarted","Data":"1f77813dfb5fcbd4a488d674088b215c3cb4bb359b19f1ce39452e3c0561d0aa"}
Sep 11 18:15:46.971471 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.971446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb" event={"ID":"62fdeac2-769b-411d-b103-9ca91d804f1f","Type":"ContainerStarted","Data":"3eddcfb513b40cb42f638e6621f10f49cdceb54ccc7ef40de9255ce060f66bb1"}
Sep 11 18:15:46.971613 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.971517    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb" event={"ID":"62fdeac2-769b-411d-b103-9ca91d804f1f","Type":"ContainerStarted","Data":"d532d8cd1f5e7d224cfb5043d48e277ec73dbb3e7f0ae4521c4d10b6b1c47296"}
Sep 11 18:15:46.971613 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.971567    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb" event={"ID":"62fdeac2-769b-411d-b103-9ca91d804f1f","Type":"ContainerStarted","Data":"6ff34ebf9b377a40ae6582b28f44f6202db51b57a9cc0e08e6053773aaba7071"}
Sep 11 18:15:46.971927 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.971889    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zdcg9"
Sep 11 18:15:46.972324 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:46.972307    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zdcg9"
Sep 11 18:15:47.193297 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:47.193275    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 18:15:47.677000 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:47.676869    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T18:15:47.193293802Z","UUID":"b20c6111-b125-472f-a045-2f7f9a530b17","Handler":null,"Name":"","Endpoint":""}
Sep 11 18:15:47.678496 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:47.678461    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 18:15:47.678596 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:47.678511    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 18:15:47.758510 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:47.758421    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:47.758510 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:47.758427    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:47.758729 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:47.758568    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:47.758729 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:47.758677    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:47.975687 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:47.975605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg" event={"ID":"8de484a7-3da4-4dd7-82da-46fe04ad4508","Type":"ContainerStarted","Data":"63cede9b5ca3f38676924f77a3570304c7ac52af8cb6a7c2f067ca0a2707bd6d"}
Sep 11 18:15:47.978370 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:47.978336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q8nvm" event={"ID":"7a3a8fa6-2a7b-4d7a-ac23-63babb814575","Type":"ContainerStarted","Data":"0191f831bf42efe4c4c41463128200fd60aa0cdf1148fa9e95f436f419bddda1"}
Sep 11 18:15:47.997386 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:47.997343    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q8nvm" podStartSLOduration=5.096938783 podStartE2EDuration="23.997330692s" totalImagesPullingTime="18.900391909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:47.996978528 +0000 UTC m=+24.738105392" watchObservedRunningTime="2026-09-11 18:15:47.997330692 +0000 UTC m=+24.738457534"
Sep 11 18:15:48.757958 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:48.757898    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:48.758086 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:48.757997    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:48.988059 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:48.988024    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb" event={"ID":"62fdeac2-769b-411d-b103-9ca91d804f1f","Type":"ContainerStarted","Data":"a27b4d592bdac4471955dc8267bc38073ee040c2c3d92af152e6ca8654af1c74"}
Sep 11 18:15:48.990511 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:48.990434    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg" event={"ID":"8de484a7-3da4-4dd7-82da-46fe04ad4508","Type":"ContainerStarted","Data":"820dc6c31c67a7f84b221c542aa6fdefb04e5745180d4d39b0c96dae7e906c08"}
Sep 11 18:15:49.016196 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:49.016094    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8xmwg" podStartSLOduration=3.63159271 podStartE2EDuration="25.016076887s" totalImagesPullingTime="21.384484177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:49.014962613 +0000 UTC m=+25.756089458" watchObservedRunningTime="2026-09-11 18:15:49.016076887 +0000 UTC m=+25.757203735"
Sep 11 18:15:49.758524 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:49.758247    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:49.758706 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:49.758262    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:49.758706 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:49.758665    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:49.758819 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:49.758795    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:50.757967 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:50.757529    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:50.757967 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:50.757689    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:51.757810 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:51.757766    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:51.757970 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:51.757879    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:51.758179 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:51.758146    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:51.758854 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:51.758268    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:51.998040 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:51.998011    2581 generic.go:356] "Generic (PLEG): container finished" podID="608a1286-9f29-437e-b113-0f4cf3015c7b" containerID="5dc9f6b40d0765f847df23d1b8c44614715aa3e0d6e10f2491fcbb15982c16f3" exitCode=0
Sep 11 18:15:51.998202 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:51.998088    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6fm6" event={"ID":"608a1286-9f29-437e-b113-0f4cf3015c7b","Type":"ContainerDied","Data":"5dc9f6b40d0765f847df23d1b8c44614715aa3e0d6e10f2491fcbb15982c16f3"}
Sep 11 18:15:52.001913 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:52.001889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb" event={"ID":"62fdeac2-769b-411d-b103-9ca91d804f1f","Type":"ContainerStarted","Data":"602beb893d697f414460fac4f301d57f0ee1ce880064a3779712961f9630c358"}
Sep 11 18:15:52.002154 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:52.002138    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:52.017462 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:52.017443    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:52.042468 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:52.042422    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb" podStartSLOduration=9.099978583 podStartE2EDuration="28.042410922s" totalImagesPullingTime="18.942432339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:52.041139475 +0000 UTC m=+28.782266322" watchObservedRunningTime="2026-09-11 18:15:52.042410922 +0000 UTC m=+28.783537764"
Sep 11 18:15:52.757518 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:52.757488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:52.757669 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:52.757624    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:53.006557 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:53.006526    2581 generic.go:356] "Generic (PLEG): container finished" podID="608a1286-9f29-437e-b113-0f4cf3015c7b" containerID="43884fae807202e9920f19700fa5b1fedb6c92fb2ed059ea43ab8196a851fe85" exitCode=0
Sep 11 18:15:53.006901 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:53.006609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6fm6" event={"ID":"608a1286-9f29-437e-b113-0f4cf3015c7b","Type":"ContainerDied","Data":"43884fae807202e9920f19700fa5b1fedb6c92fb2ed059ea43ab8196a851fe85"}
Sep 11 18:15:53.007152 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:53.007073    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:53.007152 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:53.007139    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:53.022392 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:53.022371    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:15:53.488924 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:53.488885    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9rs25"]
Sep 11 18:15:53.489080 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:53.489023    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:53.489163 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:53.489138    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:53.494708 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:53.494674    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fxtlq"]
Sep 11 18:15:53.494834 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:53.494775    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:53.495932 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:53.495905    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:53.502193 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:53.501651    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-48sl5"]
Sep 11 18:15:53.502193 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:53.501831    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:53.502193 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:53.501962    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:54.011181 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:54.011142    2581 generic.go:356] "Generic (PLEG): container finished" podID="608a1286-9f29-437e-b113-0f4cf3015c7b" containerID="8e960e9cea5b9c8a1c1748f01ca80d9779cf9cdf791143ba033ec339a53037b8" exitCode=0
Sep 11 18:15:54.011584 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:54.011219    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6fm6" event={"ID":"608a1286-9f29-437e-b113-0f4cf3015c7b","Type":"ContainerDied","Data":"8e960e9cea5b9c8a1c1748f01ca80d9779cf9cdf791143ba033ec339a53037b8"}
Sep 11 18:15:54.757433 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:54.757403    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:54.757433 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:54.757424    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:54.757696 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:54.757517    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:54.757696 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:54.757626    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:54.757696 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:54.757679    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:54.757851 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:54.757775    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:56.759262 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:56.758067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:56.759262 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:56.758108    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:56.759262 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:56.758108    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:56.759262 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:56.758208    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48sl5" podUID="db03c3a4-cfd5-443f-afe5-6e2fad4b0678"
Sep 11 18:15:56.759262 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:56.758285    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rs25" podUID="eb4f3610-8190-4056-88be-96c5087cb949"
Sep 11 18:15:56.759262 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:56.758347    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fxtlq" podUID="a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b"
Sep 11 18:15:57.374680 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:57.374644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs\") pod \"network-metrics-daemon-48sl5\" (UID: \"db03c3a4-cfd5-443f-afe5-6e2fad4b0678\") " pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:57.374849 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:57.374768    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:57.374905 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:57.374851    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs podName:db03c3a4-cfd5-443f-afe5-6e2fad4b0678 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:29.374818614 +0000 UTC m=+66.115945435 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs") pod "network-metrics-daemon-48sl5" (UID: "db03c3a4-cfd5-443f-afe5-6e2fad4b0678") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:57.475020 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:57.474986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grb7r\" (UniqueName: \"kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r\") pod \"network-check-target-fxtlq\" (UID: \"a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b\") " pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:57.475020 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:57.475025    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret\") pod \"global-pull-secret-syncer-9rs25\" (UID: \"eb4f3610-8190-4056-88be-96c5087cb949\") " pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:57.475236 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:57.475135    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:57.475236 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:57.475143    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:57.475236 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:57.475166    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:57.475236 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:57.475177    2581 projected.go:196] Error preparing data for projected volume kube-api-access-grb7r for pod openshift-network-diagnostics/network-check-target-fxtlq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:57.475236 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:57.475195    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret podName:eb4f3610-8190-4056-88be-96c5087cb949 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:29.475181494 +0000 UTC m=+66.216308315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret") pod "global-pull-secret-syncer-9rs25" (UID: "eb4f3610-8190-4056-88be-96c5087cb949") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:57.475236 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:57.475226    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r podName:a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b nodeName:}" failed. No retries permitted until 2026-09-11 18:16:29.475211513 +0000 UTC m=+66.216338353 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-grb7r" (UniqueName: "kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r") pod "network-check-target-fxtlq" (UID: "a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:58.518973 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.518937    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-166.ec2.internal" event="NodeReady"
Sep 11 18:15:58.519427 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.519064    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 18:15:58.565140 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.565099    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lck25"]
Sep 11 18:15:58.580981 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.580946    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gg6sb"]
Sep 11 18:15:58.581134 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.581066    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lck25"
Sep 11 18:15:58.583355 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.583329    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b9ltz\""
Sep 11 18:15:58.583503 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.583401    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 18:15:58.583503 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.583424    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 18:15:58.600386 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.600361    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lck25"]
Sep 11 18:15:58.600530 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.600401    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gg6sb"]
Sep 11 18:15:58.600530 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.600463    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gg6sb"
Sep 11 18:15:58.602402 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.602386    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 18:15:58.602535 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.602518    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 18:15:58.602606 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.602553    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 18:15:58.602606 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.602588    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kjt2v\""
Sep 11 18:15:58.683341 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.683317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a008ce-8cd5-4753-a3ad-7337832bf965-config-volume\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:15:58.683428 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.683349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:15:58.683428 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.683393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb95q\" (UniqueName: \"kubernetes.io/projected/e8a008ce-8cd5-4753-a3ad-7337832bf965-kube-api-access-pb95q\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:15:58.683428 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.683418    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a008ce-8cd5-4753-a3ad-7337832bf965-tmp-dir\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:15:58.757617 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.757564    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:15:58.757693 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.757661    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:15:58.757693 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.757685    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:15:58.759821 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.759799    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 18:15:58.759931 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.759822    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-78r7v\""
Sep 11 18:15:58.759931 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.759799    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 18:15:58.759931 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.759804    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 18:15:58.760138 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.760123    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-28rql\""
Sep 11 18:15:58.760174 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.760154    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 18:15:58.784467 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.784441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a008ce-8cd5-4753-a3ad-7337832bf965-tmp-dir\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:15:58.784591 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.784471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6eb04a4-7bb3-4a28-90e8-1e09e88a35ba-cert\") pod \"ingress-canary-gg6sb\" (UID: \"a6eb04a4-7bb3-4a28-90e8-1e09e88a35ba\") " pod="openshift-ingress-canary/ingress-canary-gg6sb"
Sep 11 18:15:58.784591 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.784530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a008ce-8cd5-4753-a3ad-7337832bf965-config-volume\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:15:58.784591 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.784546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:15:58.784591 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.784578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb95q\" (UniqueName: \"kubernetes.io/projected/e8a008ce-8cd5-4753-a3ad-7337832bf965-kube-api-access-pb95q\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:15:58.784801 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.784593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmzns\" (UniqueName: \"kubernetes.io/projected/a6eb04a4-7bb3-4a28-90e8-1e09e88a35ba-kube-api-access-qmzns\") pod \"ingress-canary-gg6sb\" (UID: \"a6eb04a4-7bb3-4a28-90e8-1e09e88a35ba\") " pod="openshift-ingress-canary/ingress-canary-gg6sb"
Sep 11 18:15:58.784801 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:58.784699    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:15:58.784801 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:58.784773    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls podName:e8a008ce-8cd5-4753-a3ad-7337832bf965 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:59.284756918 +0000 UTC m=+36.025883746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls") pod "dns-default-lck25" (UID: "e8a008ce-8cd5-4753-a3ad-7337832bf965") : secret "dns-default-metrics-tls" not found
Sep 11 18:15:58.784913 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.784805    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8a008ce-8cd5-4753-a3ad-7337832bf965-tmp-dir\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:15:58.785091 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.785074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e8a008ce-8cd5-4753-a3ad-7337832bf965-config-volume\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:15:58.795989 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.795967    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb95q\" (UniqueName: \"kubernetes.io/projected/e8a008ce-8cd5-4753-a3ad-7337832bf965-kube-api-access-pb95q\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:15:58.885151 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.885128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmzns\" (UniqueName: \"kubernetes.io/projected/a6eb04a4-7bb3-4a28-90e8-1e09e88a35ba-kube-api-access-qmzns\") pod \"ingress-canary-gg6sb\" (UID: \"a6eb04a4-7bb3-4a28-90e8-1e09e88a35ba\") " pod="openshift-ingress-canary/ingress-canary-gg6sb"
Sep 11 18:15:58.885265 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.885165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6eb04a4-7bb3-4a28-90e8-1e09e88a35ba-cert\") pod \"ingress-canary-gg6sb\" (UID: \"a6eb04a4-7bb3-4a28-90e8-1e09e88a35ba\") " pod="openshift-ingress-canary/ingress-canary-gg6sb"
Sep 11 18:15:58.887351 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.887334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a6eb04a4-7bb3-4a28-90e8-1e09e88a35ba-cert\") pod \"ingress-canary-gg6sb\" (UID: \"a6eb04a4-7bb3-4a28-90e8-1e09e88a35ba\") " pod="openshift-ingress-canary/ingress-canary-gg6sb"
Sep 11 18:15:58.893595 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.893575    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmzns\" (UniqueName: \"kubernetes.io/projected/a6eb04a4-7bb3-4a28-90e8-1e09e88a35ba-kube-api-access-qmzns\") pod \"ingress-canary-gg6sb\" (UID: \"a6eb04a4-7bb3-4a28-90e8-1e09e88a35ba\") " pod="openshift-ingress-canary/ingress-canary-gg6sb"
Sep 11 18:15:58.908357 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:58.908337    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gg6sb"
Sep 11 18:15:59.025746 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:59.025682    2581 generic.go:356] "Generic (PLEG): container finished" podID="608a1286-9f29-437e-b113-0f4cf3015c7b" containerID="917a7672940597be10599035f463adaeb14c1eaebf2d9b31c63d01f8a9961fb4" exitCode=0
Sep 11 18:15:59.025746 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:59.025742    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6fm6" event={"ID":"608a1286-9f29-437e-b113-0f4cf3015c7b","Type":"ContainerDied","Data":"917a7672940597be10599035f463adaeb14c1eaebf2d9b31c63d01f8a9961fb4"}
Sep 11 18:15:59.211753 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:59.211596    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gg6sb"]
Sep 11 18:15:59.249217 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:15:59.249193    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6eb04a4_7bb3_4a28_90e8_1e09e88a35ba.slice/crio-b72c430853a3688cabed01fa75d192b0f3620f14a727b2629d61df80b57fee28 WatchSource:0}: Error finding container b72c430853a3688cabed01fa75d192b0f3620f14a727b2629d61df80b57fee28: Status 404 returned error can't find the container with id b72c430853a3688cabed01fa75d192b0f3620f14a727b2629d61df80b57fee28
Sep 11 18:15:59.288152 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:15:59.288107    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:15:59.288228 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:59.288220    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:15:59.288271 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:15:59.288262    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls podName:e8a008ce-8cd5-4753-a3ad-7337832bf965 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:00.288249097 +0000 UTC m=+37.029375918 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls") pod "dns-default-lck25" (UID: "e8a008ce-8cd5-4753-a3ad-7337832bf965") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:00.031754 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:00.031722    2581 generic.go:356] "Generic (PLEG): container finished" podID="608a1286-9f29-437e-b113-0f4cf3015c7b" containerID="96b542bb019a953392c9c1d8b4883211d73843e009cd752dbd8b6b79f00924a3" exitCode=0
Sep 11 18:16:00.032634 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:00.031802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6fm6" event={"ID":"608a1286-9f29-437e-b113-0f4cf3015c7b","Type":"ContainerDied","Data":"96b542bb019a953392c9c1d8b4883211d73843e009cd752dbd8b6b79f00924a3"}
Sep 11 18:16:00.033202 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:00.033175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gg6sb" event={"ID":"a6eb04a4-7bb3-4a28-90e8-1e09e88a35ba","Type":"ContainerStarted","Data":"b72c430853a3688cabed01fa75d192b0f3620f14a727b2629d61df80b57fee28"}
Sep 11 18:16:00.297159 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:00.297076    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:16:00.297320 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:00.297220    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:00.297320 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:00.297281    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls podName:e8a008ce-8cd5-4753-a3ad-7337832bf965 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:02.29726397 +0000 UTC m=+39.038390808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls") pod "dns-default-lck25" (UID: "e8a008ce-8cd5-4753-a3ad-7337832bf965") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:01.039745 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:01.039707    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-p6fm6" event={"ID":"608a1286-9f29-437e-b113-0f4cf3015c7b","Type":"ContainerStarted","Data":"0a37aa5273a1ed2ddcc8b940f0d8696b52a0b87a3c4b318e2d1fe68803bfb3e2"}
Sep 11 18:16:01.062784 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:01.062726    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-p6fm6" podStartSLOduration=8.067345215 podStartE2EDuration="37.062684064s" totalImagesPullingTime="28.995338849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:01.061103662 +0000 UTC m=+37.802230509" watchObservedRunningTime="2026-09-11 18:16:01.062684064 +0000 UTC m=+37.803810924"
Sep 11 18:16:02.043561 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:02.043509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gg6sb" event={"ID":"a6eb04a4-7bb3-4a28-90e8-1e09e88a35ba","Type":"ContainerStarted","Data":"51b3f7ecbff623f45f8b218688dc2d3a16e7c06ef4fa414013fa2e542618e2cc"}
Sep 11 18:16:02.061778 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:02.061733    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gg6sb" podStartSLOduration=1.760482034 podStartE2EDuration="4.061719627s" totalImagesPullingTime="2.301237593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:02.061513169 +0000 UTC m=+38.802640017" watchObservedRunningTime="2026-09-11 18:16:02.061719627 +0000 UTC m=+38.802846470"
Sep 11 18:16:02.313437 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:02.313367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:16:02.313602 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:02.313510    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:02.313602 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:02.313568    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls podName:e8a008ce-8cd5-4753-a3ad-7337832bf965 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:06.313552515 +0000 UTC m=+43.054679337 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls") pod "dns-default-lck25" (UID: "e8a008ce-8cd5-4753-a3ad-7337832bf965") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:06.337094 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:06.337050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:16:06.337446 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:06.337205    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:06.337446 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:06.337281    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls podName:e8a008ce-8cd5-4753-a3ad-7337832bf965 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:14.337264607 +0000 UTC m=+51.078391428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls") pod "dns-default-lck25" (UID: "e8a008ce-8cd5-4753-a3ad-7337832bf965") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:14.390453 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:14.390412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:16:14.390874 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:14.390586    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:14.390874 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:14.390644    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls podName:e8a008ce-8cd5-4753-a3ad-7337832bf965 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:30.39062991 +0000 UTC m=+67.131756731 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls") pod "dns-default-lck25" (UID: "e8a008ce-8cd5-4753-a3ad-7337832bf965") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:25.025147 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:25.025119    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mx4jb"
Sep 11 18:16:29.393372 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:29.393336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs\") pod \"network-metrics-daemon-48sl5\" (UID: \"db03c3a4-cfd5-443f-afe5-6e2fad4b0678\") " pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:16:29.395788 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:29.395770    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 18:16:29.404405 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:29.404388    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 18:16:29.404496 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:29.404439    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs podName:db03c3a4-cfd5-443f-afe5-6e2fad4b0678 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:33.404424812 +0000 UTC m=+130.145551638 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs") pod "network-metrics-daemon-48sl5" (UID: "db03c3a4-cfd5-443f-afe5-6e2fad4b0678") : secret "metrics-daemon-secret" not found
Sep 11 18:16:29.494538 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:29.494507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grb7r\" (UniqueName: \"kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r\") pod \"network-check-target-fxtlq\" (UID: \"a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b\") " pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:16:29.494660 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:29.494544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret\") pod \"global-pull-secret-syncer-9rs25\" (UID: \"eb4f3610-8190-4056-88be-96c5087cb949\") " pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:16:29.496797 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:29.496775    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 18:16:29.496908 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:29.496796    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 18:16:29.506859 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:29.506837    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 18:16:29.508862 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:29.508846    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eb4f3610-8190-4056-88be-96c5087cb949-original-pull-secret\") pod \"global-pull-secret-syncer-9rs25\" (UID: \"eb4f3610-8190-4056-88be-96c5087cb949\") " pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:16:29.518163 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:29.518136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grb7r\" (UniqueName: \"kubernetes.io/projected/a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b-kube-api-access-grb7r\") pod \"network-check-target-fxtlq\" (UID: \"a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b\") " pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:16:29.674107 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:29.674048    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-78r7v\""
Sep 11 18:16:29.676661 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:29.676648    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rs25"
Sep 11 18:16:29.682877 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:29.682860    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:16:29.971900 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:29.971819    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9rs25"]
Sep 11 18:16:29.975530 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:16:29.975467    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4f3610_8190_4056_88be_96c5087cb949.slice/crio-7ad4201ff6f5c4e21fff4804ea7f1f1d847fe94db800e46d9560bcbbc3251beb WatchSource:0}: Error finding container 7ad4201ff6f5c4e21fff4804ea7f1f1d847fe94db800e46d9560bcbbc3251beb: Status 404 returned error can't find the container with id 7ad4201ff6f5c4e21fff4804ea7f1f1d847fe94db800e46d9560bcbbc3251beb
Sep 11 18:16:29.996592 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:29.996571    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fxtlq"]
Sep 11 18:16:29.999105 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:16:29.999079    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d38c0a_ba8d_44ca_9b1e_ac4f3b4c243b.slice/crio-1463f938ec6b856008b7354b38227d7abb38360680108738fde3685c5aea1c54 WatchSource:0}: Error finding container 1463f938ec6b856008b7354b38227d7abb38360680108738fde3685c5aea1c54: Status 404 returned error can't find the container with id 1463f938ec6b856008b7354b38227d7abb38360680108738fde3685c5aea1c54
Sep 11 18:16:30.108363 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:30.108325    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fxtlq" event={"ID":"a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b","Type":"ContainerStarted","Data":"1463f938ec6b856008b7354b38227d7abb38360680108738fde3685c5aea1c54"}
Sep 11 18:16:30.109395 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:30.109371    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9rs25" event={"ID":"eb4f3610-8190-4056-88be-96c5087cb949","Type":"ContainerStarted","Data":"7ad4201ff6f5c4e21fff4804ea7f1f1d847fe94db800e46d9560bcbbc3251beb"}
Sep 11 18:16:30.401201 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:30.401169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:16:30.401634 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:30.401311    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:30.401634 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:30.401369    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls podName:e8a008ce-8cd5-4753-a3ad-7337832bf965 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:02.40135595 +0000 UTC m=+99.142482783 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls") pod "dns-default-lck25" (UID: "e8a008ce-8cd5-4753-a3ad-7337832bf965") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:35.126239 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.126202    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fxtlq" event={"ID":"a5d38c0a-ba8d-44ca-9b1e-ac4f3b4c243b","Type":"ContainerStarted","Data":"4a81335648f91c7f2d510d1e0a5d5dedb38dd3c2c9f7d52a2274e6c5d1a08491"}
Sep 11 18:16:35.126704 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.126320    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:16:35.127753 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.127730    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9rs25" event={"ID":"eb4f3610-8190-4056-88be-96c5087cb949","Type":"ContainerStarted","Data":"8f1334d66180c469201ea1932937b08b69f2da6a1113437b949f0b132bc1f5a5"}
Sep 11 18:16:35.152414 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.152372    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fxtlq" podStartSLOduration=66.722643675 podStartE2EDuration="1m11.152358404s" totalImagesPullingTime="4.429714729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:35.151723636 +0000 UTC m=+71.892850479" watchObservedRunningTime="2026-09-11 18:16:35.152358404 +0000 UTC m=+71.893485274"
Sep 11 18:16:35.173724 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.173687    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9rs25" podStartSLOduration=66.717060984 podStartE2EDuration="1m11.173658455s" totalImagesPullingTime="4.456597471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:35.172717132 +0000 UTC m=+71.913843976" watchObservedRunningTime="2026-09-11 18:16:35.173658455 +0000 UTC m=+71.914802048"
Sep 11 18:16:35.225508 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.225468    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7dcc55bd66-526gs"]
Sep 11 18:16:35.227217 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.227205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:35.229676 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.229659    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 18:16:35.230541 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.230527    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 11 18:16:35.231701 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.231682    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 11 18:16:35.243576 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.243560    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 11 18:16:35.248011 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.247993    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zklk4\""
Sep 11 18:16:35.257031 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.257012    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 11 18:16:35.257130 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.257017    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 18:16:35.284035 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.284015    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7dcc55bd66-526gs"]
Sep 11 18:16:35.333881 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.333855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:35.333988 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.333884    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:35.333988 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.333906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks6bs\" (UniqueName: \"kubernetes.io/projected/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-kube-api-access-ks6bs\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:35.333988 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.333954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-default-certificate\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:35.334089 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.334024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-stats-auth\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:35.412801 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.412740    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-hdczs"]
Sep 11 18:16:35.414541 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.414527    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hdczs"
Sep 11 18:16:35.416401 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.416380    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-787c864788-jdqjl"]
Sep 11 18:16:35.416806 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.416791    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-n49vz\""
Sep 11 18:16:35.417904 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.417891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.419448 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.419431    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 18:16:35.419856 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.419839    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 18:16:35.419856 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.419849    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 18:16:35.420234 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.420209    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vxws4\""
Sep 11 18:16:35.424679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.424659    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-hdczs"]
Sep 11 18:16:35.425314 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.425288    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 18:16:35.435255 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.435236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-stats-auth\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:35.435343 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.435268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:35.435343 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.435285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:35.435343 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.435303    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks6bs\" (UniqueName: \"kubernetes.io/projected/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-kube-api-access-ks6bs\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:35.435343 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.435324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-default-certificate\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:35.435568 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:35.435380    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 18:16:35.435568 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:35.435419    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle podName:c230cd27-7b95-48a6-8cd8-9f10f8a87d8a nodeName:}" failed. No retries permitted until 2026-09-11 18:16:35.935399287 +0000 UTC m=+72.676526126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle") pod "router-default-7dcc55bd66-526gs" (UID: "c230cd27-7b95-48a6-8cd8-9f10f8a87d8a") : configmap references non-existent config key: service-ca.crt
Sep 11 18:16:35.435568 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:35.435444    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs podName:c230cd27-7b95-48a6-8cd8-9f10f8a87d8a nodeName:}" failed. No retries permitted until 2026-09-11 18:16:35.935433056 +0000 UTC m=+72.676559884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs") pod "router-default-7dcc55bd66-526gs" (UID: "c230cd27-7b95-48a6-8cd8-9f10f8a87d8a") : secret "router-metrics-certs-default" not found
Sep 11 18:16:35.437004 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.436982    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-787c864788-jdqjl"]
Sep 11 18:16:35.437828 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.437806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-stats-auth\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:35.438006 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.437987    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-default-certificate\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:35.450161 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.450141    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks6bs\" (UniqueName: \"kubernetes.io/projected/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-kube-api-access-ks6bs\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:35.536516 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.536467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-certificates\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.536622 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.536521    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1728312-8558-4aec-86f6-c48c7dfdaffa-trusted-ca\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.536622 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.536551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1728312-8558-4aec-86f6-c48c7dfdaffa-installation-pull-secrets\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.536622 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.536576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-bound-sa-token\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.536759 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.536622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4gv7\" (UniqueName: \"kubernetes.io/projected/991e7d00-8b86-4020-b96a-e2d9484147f7-kube-api-access-q4gv7\") pod \"network-check-source-6f8b768455-hdczs\" (UID: \"991e7d00-8b86-4020-b96a-e2d9484147f7\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-hdczs"
Sep 11 18:16:35.536759 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.536668    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1728312-8558-4aec-86f6-c48c7dfdaffa-ca-trust-extracted\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.536824 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.536764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.536824 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.536788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfd5z\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-kube-api-access-lfd5z\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.536824 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.536807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1728312-8558-4aec-86f6-c48c7dfdaffa-image-registry-private-configuration\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.637995 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.637969    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-certificates\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.638086 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.637999    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1728312-8558-4aec-86f6-c48c7dfdaffa-trusted-ca\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.638086 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.638016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1728312-8558-4aec-86f6-c48c7dfdaffa-installation-pull-secrets\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.638086 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.638033    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-bound-sa-token\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.638237 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.638164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4gv7\" (UniqueName: \"kubernetes.io/projected/991e7d00-8b86-4020-b96a-e2d9484147f7-kube-api-access-q4gv7\") pod \"network-check-source-6f8b768455-hdczs\" (UID: \"991e7d00-8b86-4020-b96a-e2d9484147f7\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-hdczs"
Sep 11 18:16:35.638237 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.638218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1728312-8558-4aec-86f6-c48c7dfdaffa-ca-trust-extracted\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.638346 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.638289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.638346 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.638314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfd5z\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-kube-api-access-lfd5z\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.638463 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.638343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1728312-8558-4aec-86f6-c48c7dfdaffa-image-registry-private-configuration\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.638463 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:35.638394    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:16:35.638463 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:35.638414    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787c864788-jdqjl: secret "image-registry-tls" not found
Sep 11 18:16:35.638668 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:35.638529    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls podName:e1728312-8558-4aec-86f6-c48c7dfdaffa nodeName:}" failed. No retries permitted until 2026-09-11 18:16:36.13850983 +0000 UTC m=+72.879636657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls") pod "image-registry-787c864788-jdqjl" (UID: "e1728312-8558-4aec-86f6-c48c7dfdaffa") : secret "image-registry-tls" not found
Sep 11 18:16:35.638668 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.638637    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1728312-8558-4aec-86f6-c48c7dfdaffa-ca-trust-extracted\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.638796 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.638665    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-certificates\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.639005 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.638988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1728312-8558-4aec-86f6-c48c7dfdaffa-trusted-ca\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.640467 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.640445    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1728312-8558-4aec-86f6-c48c7dfdaffa-installation-pull-secrets\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.640648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.640630    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1728312-8558-4aec-86f6-c48c7dfdaffa-image-registry-private-configuration\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.648769 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.648749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-bound-sa-token\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.649117 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.649097    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4gv7\" (UniqueName: \"kubernetes.io/projected/991e7d00-8b86-4020-b96a-e2d9484147f7-kube-api-access-q4gv7\") pod \"network-check-source-6f8b768455-hdczs\" (UID: \"991e7d00-8b86-4020-b96a-e2d9484147f7\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-hdczs"
Sep 11 18:16:35.649214 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.649198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfd5z\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-kube-api-access-lfd5z\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:35.725472 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.725427    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hdczs"
Sep 11 18:16:35.940826 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.940786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:35.940988 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.940830    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:35.940988 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:35.940957    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 18:16:35.941059 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:35.941015    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs podName:c230cd27-7b95-48a6-8cd8-9f10f8a87d8a nodeName:}" failed. No retries permitted until 2026-09-11 18:16:36.940998214 +0000 UTC m=+73.682125037 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs") pod "router-default-7dcc55bd66-526gs" (UID: "c230cd27-7b95-48a6-8cd8-9f10f8a87d8a") : secret "router-metrics-certs-default" not found
Sep 11 18:16:35.941059 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:35.941029    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle podName:c230cd27-7b95-48a6-8cd8-9f10f8a87d8a nodeName:}" failed. No retries permitted until 2026-09-11 18:16:36.941023702 +0000 UTC m=+73.682150523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle") pod "router-default-7dcc55bd66-526gs" (UID: "c230cd27-7b95-48a6-8cd8-9f10f8a87d8a") : configmap references non-existent config key: service-ca.crt
Sep 11 18:16:35.994095 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:35.994031    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-hdczs"]
Sep 11 18:16:35.996978 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:16:35.996953    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991e7d00_8b86_4020_b96a_e2d9484147f7.slice/crio-a5e995b4b0bbd1304c011004b99d06c9daa8c457d26672afd0d247e408e07be6 WatchSource:0}: Error finding container a5e995b4b0bbd1304c011004b99d06c9daa8c457d26672afd0d247e408e07be6: Status 404 returned error can't find the container with id a5e995b4b0bbd1304c011004b99d06c9daa8c457d26672afd0d247e408e07be6
Sep 11 18:16:36.131817 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:36.131775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hdczs" event={"ID":"991e7d00-8b86-4020-b96a-e2d9484147f7","Type":"ContainerStarted","Data":"46a0fac867eb074de55002762d9e1adcfe71b8aefa9b3480b0b57dbe21e744dd"}
Sep 11 18:16:36.132220 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:36.131829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hdczs" event={"ID":"991e7d00-8b86-4020-b96a-e2d9484147f7","Type":"ContainerStarted","Data":"a5e995b4b0bbd1304c011004b99d06c9daa8c457d26672afd0d247e408e07be6"}
Sep 11 18:16:36.142710 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:36.142683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:36.142841 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:36.142780    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:16:36.142841 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:36.142789    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787c864788-jdqjl: secret "image-registry-tls" not found
Sep 11 18:16:36.142841 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:36.142832    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls podName:e1728312-8558-4aec-86f6-c48c7dfdaffa nodeName:}" failed. No retries permitted until 2026-09-11 18:16:37.142820229 +0000 UTC m=+73.883947049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls") pod "image-registry-787c864788-jdqjl" (UID: "e1728312-8558-4aec-86f6-c48c7dfdaffa") : secret "image-registry-tls" not found
Sep 11 18:16:36.146883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:36.146842    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-hdczs" podStartSLOduration=1.146827243 podStartE2EDuration="1.146827243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:36.146063585 +0000 UTC m=+72.887190427" watchObservedRunningTime="2026-09-11 18:16:36.146827243 +0000 UTC m=+72.887954086"
Sep 11 18:16:36.947907 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:36.947866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:36.947907 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:36.947908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:36.948110 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:36.948017    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 18:16:36.948110 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:36.948042    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle podName:c230cd27-7b95-48a6-8cd8-9f10f8a87d8a nodeName:}" failed. No retries permitted until 2026-09-11 18:16:38.94802412 +0000 UTC m=+75.689150957 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle") pod "router-default-7dcc55bd66-526gs" (UID: "c230cd27-7b95-48a6-8cd8-9f10f8a87d8a") : configmap references non-existent config key: service-ca.crt
Sep 11 18:16:36.948110 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:36.948060    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs podName:c230cd27-7b95-48a6-8cd8-9f10f8a87d8a nodeName:}" failed. No retries permitted until 2026-09-11 18:16:38.948049574 +0000 UTC m=+75.689176410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs") pod "router-default-7dcc55bd66-526gs" (UID: "c230cd27-7b95-48a6-8cd8-9f10f8a87d8a") : secret "router-metrics-certs-default" not found
Sep 11 18:16:37.149565 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:37.149530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:37.149921 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:37.149645    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:16:37.149921 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:37.149655    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787c864788-jdqjl: secret "image-registry-tls" not found
Sep 11 18:16:37.149921 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:37.149700    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls podName:e1728312-8558-4aec-86f6-c48c7dfdaffa nodeName:}" failed. No retries permitted until 2026-09-11 18:16:39.149687107 +0000 UTC m=+75.890813927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls") pod "image-registry-787c864788-jdqjl" (UID: "e1728312-8558-4aec-86f6-c48c7dfdaffa") : secret "image-registry-tls" not found
Sep 11 18:16:38.960940 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:38.960905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:38.960940 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:38.960939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:38.961339 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:38.961057    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle podName:c230cd27-7b95-48a6-8cd8-9f10f8a87d8a nodeName:}" failed. No retries permitted until 2026-09-11 18:16:42.961044519 +0000 UTC m=+79.702171340 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle") pod "router-default-7dcc55bd66-526gs" (UID: "c230cd27-7b95-48a6-8cd8-9f10f8a87d8a") : configmap references non-existent config key: service-ca.crt
Sep 11 18:16:38.961339 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:38.961068    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 18:16:38.961339 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:38.961129    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs podName:c230cd27-7b95-48a6-8cd8-9f10f8a87d8a nodeName:}" failed. No retries permitted until 2026-09-11 18:16:42.96111207 +0000 UTC m=+79.702238911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs") pod "router-default-7dcc55bd66-526gs" (UID: "c230cd27-7b95-48a6-8cd8-9f10f8a87d8a") : secret "router-metrics-certs-default" not found
Sep 11 18:16:39.162180 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:39.162149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:39.162328 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:39.162297    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:16:39.162328 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:39.162314    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787c864788-jdqjl: secret "image-registry-tls" not found
Sep 11 18:16:39.162397 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:39.162368    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls podName:e1728312-8558-4aec-86f6-c48c7dfdaffa nodeName:}" failed. No retries permitted until 2026-09-11 18:16:43.16235394 +0000 UTC m=+79.903480761 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls") pod "image-registry-787c864788-jdqjl" (UID: "e1728312-8558-4aec-86f6-c48c7dfdaffa") : secret "image-registry-tls" not found
Sep 11 18:16:40.990807 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:40.990779    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bv5j5"]
Sep 11 18:16:40.993597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:40.993572    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:40.995564 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:40.995545    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 18:16:40.995687 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:40.995569    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 18:16:40.996318 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:40.996297    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 18:16:40.996318 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:40.996315    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fpnpf\""
Sep 11 18:16:40.996458 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:40.996298    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 18:16:41.003728 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:41.003706    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bv5j5"]
Sep 11 18:16:41.074987 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:41.074965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96e20d2c-23a6-4e42-a459-e680cddc8572-crio-socket\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:41.075137 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:41.074992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:41.075137 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:41.075054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:41.075137 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:41.075112    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96e20d2c-23a6-4e42-a459-e680cddc8572-data-volume\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:41.075249 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:41.075135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hv96\" (UniqueName: \"kubernetes.io/projected/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-api-access-6hv96\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:41.175384 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:41.175355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96e20d2c-23a6-4e42-a459-e680cddc8572-data-volume\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:41.175384 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:41.175386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hv96\" (UniqueName: \"kubernetes.io/projected/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-api-access-6hv96\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:41.175562 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:41.175411    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96e20d2c-23a6-4e42-a459-e680cddc8572-crio-socket\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:41.175562 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:41.175433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:41.175562 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:41.175521    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:41.175659 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:41.175571    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:16:41.175659 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:41.175595    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:41.175659 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:41.175521    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96e20d2c-23a6-4e42-a459-e680cddc8572-crio-socket\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:41.175659 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:41.175628    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm podName:96e20d2c-23a6-4e42-a459-e680cddc8572 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:41.675611645 +0000 UTC m=+78.416738466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bv5j5" (UID: "96e20d2c-23a6-4e42-a459-e680cddc8572") : configmap "kube-rbac-proxy" not found
Sep 11 18:16:41.175659 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:41.175651    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls podName:96e20d2c-23a6-4e42-a459-e680cddc8572 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:41.675633853 +0000 UTC m=+78.416760689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bv5j5" (UID: "96e20d2c-23a6-4e42-a459-e680cddc8572") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:41.175834 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:41.175808    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96e20d2c-23a6-4e42-a459-e680cddc8572-data-volume\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:41.187671 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:41.187650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hv96\" (UniqueName: \"kubernetes.io/projected/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-api-access-6hv96\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:41.679466 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:41.679421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:41.679657 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:41.679506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:41.679657 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:41.679561    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:16:41.679657 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:41.679594    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:41.679657 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:41.679627    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm podName:96e20d2c-23a6-4e42-a459-e680cddc8572 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:42.679610604 +0000 UTC m=+79.420737425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bv5j5" (UID: "96e20d2c-23a6-4e42-a459-e680cddc8572") : configmap "kube-rbac-proxy" not found
Sep 11 18:16:41.679657 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:41.679640    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls podName:96e20d2c-23a6-4e42-a459-e680cddc8572 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:42.679634807 +0000 UTC m=+79.420761628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bv5j5" (UID: "96e20d2c-23a6-4e42-a459-e680cddc8572") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:42.686315 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:42.686276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:42.686808 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:42.686391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:42.686808 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:42.686451    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:42.686808 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:42.686513    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:16:42.686808 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:42.686535    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls podName:96e20d2c-23a6-4e42-a459-e680cddc8572 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:44.686518773 +0000 UTC m=+81.427645624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bv5j5" (UID: "96e20d2c-23a6-4e42-a459-e680cddc8572") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:42.686808 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:42.686567    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm podName:96e20d2c-23a6-4e42-a459-e680cddc8572 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:44.686550174 +0000 UTC m=+81.427676998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bv5j5" (UID: "96e20d2c-23a6-4e42-a459-e680cddc8572") : configmap "kube-rbac-proxy" not found
Sep 11 18:16:42.748564 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:42.748541    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b9dfv_68330d27-5afb-44d1-96a2-774104ee6c4c/dns-node-resolver/0.log"
Sep 11 18:16:42.988590 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:42.988505    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:42.988590 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:42.988544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:42.988761 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:42.988627    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 11 18:16:42.988761 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:42.988687    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle podName:c230cd27-7b95-48a6-8cd8-9f10f8a87d8a nodeName:}" failed. No retries permitted until 2026-09-11 18:16:50.988674289 +0000 UTC m=+87.729801109 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle") pod "router-default-7dcc55bd66-526gs" (UID: "c230cd27-7b95-48a6-8cd8-9f10f8a87d8a") : configmap references non-existent config key: service-ca.crt
Sep 11 18:16:42.988761 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:42.988700    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs podName:c230cd27-7b95-48a6-8cd8-9f10f8a87d8a nodeName:}" failed. No retries permitted until 2026-09-11 18:16:50.988694808 +0000 UTC m=+87.729821629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs") pod "router-default-7dcc55bd66-526gs" (UID: "c230cd27-7b95-48a6-8cd8-9f10f8a87d8a") : secret "router-metrics-certs-default" not found
Sep 11 18:16:43.190204 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:43.190171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:43.190360 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:43.190321    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:16:43.190360 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:43.190337    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787c864788-jdqjl: secret "image-registry-tls" not found
Sep 11 18:16:43.190432 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:43.190392    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls podName:e1728312-8558-4aec-86f6-c48c7dfdaffa nodeName:}" failed. No retries permitted until 2026-09-11 18:16:51.190375333 +0000 UTC m=+87.931502154 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls") pod "image-registry-787c864788-jdqjl" (UID: "e1728312-8558-4aec-86f6-c48c7dfdaffa") : secret "image-registry-tls" not found
Sep 11 18:16:44.149335 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:44.149308    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xvv2h_f9719c07-ac09-4ed0-a4da-515e4d30fe2c/node-ca/0.log"
Sep 11 18:16:44.549183 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:44.549105    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gg6sb_a6eb04a4-7bb3-4a28-90e8-1e09e88a35ba/serve-healthcheck-canary/0.log"
Sep 11 18:16:44.701962 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:44.701926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:44.702155 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:44.702021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:44.702155 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:44.702075    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:44.702155 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:44.702091    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:16:44.702155 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:44.702142    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls podName:96e20d2c-23a6-4e42-a459-e680cddc8572 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:48.702126294 +0000 UTC m=+85.443253125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bv5j5" (UID: "96e20d2c-23a6-4e42-a459-e680cddc8572") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:44.702155 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:44.702157    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm podName:96e20d2c-23a6-4e42-a459-e680cddc8572 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:48.702149565 +0000 UTC m=+85.443276386 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bv5j5" (UID: "96e20d2c-23a6-4e42-a459-e680cddc8572") : configmap "kube-rbac-proxy" not found
Sep 11 18:16:48.731387 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:48.731353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:48.731766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:48.731432    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:48.731766 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:48.731515    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:48.731766 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:48.731533    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:16:48.731766 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:48.731588    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls podName:96e20d2c-23a6-4e42-a459-e680cddc8572 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:56.731570091 +0000 UTC m=+93.472696930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bv5j5" (UID: "96e20d2c-23a6-4e42-a459-e680cddc8572") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:16:48.731766 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:48.731605    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm podName:96e20d2c-23a6-4e42-a459-e680cddc8572 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:56.731596923 +0000 UTC m=+93.472723744 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bv5j5" (UID: "96e20d2c-23a6-4e42-a459-e680cddc8572") : configmap "kube-rbac-proxy" not found
Sep 11 18:16:51.046151 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:51.046113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:51.046151 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:51.046160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:51.046566 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:16:51.046370    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle podName:c230cd27-7b95-48a6-8cd8-9f10f8a87d8a nodeName:}" failed. No retries permitted until 2026-09-11 18:17:07.046353023 +0000 UTC m=+103.787479845 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle") pod "router-default-7dcc55bd66-526gs" (UID: "c230cd27-7b95-48a6-8cd8-9f10f8a87d8a") : configmap references non-existent config key: service-ca.crt
Sep 11 18:16:51.048642 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:51.048624    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-metrics-certs\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:16:51.247852 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:51.247794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:51.250160 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:51.250137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls\") pod \"image-registry-787c864788-jdqjl\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") " pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:51.330316 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:51.330265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:51.603440 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:51.603364    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-787c864788-jdqjl"]
Sep 11 18:16:51.606305 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:16:51.606278    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1728312_8558_4aec_86f6_c48c7dfdaffa.slice/crio-c7e5d03791724c431e50fa18546ce379f1a9e61385b23a99e40d7780d4c091e7 WatchSource:0}: Error finding container c7e5d03791724c431e50fa18546ce379f1a9e61385b23a99e40d7780d4c091e7: Status 404 returned error can't find the container with id c7e5d03791724c431e50fa18546ce379f1a9e61385b23a99e40d7780d4c091e7
Sep 11 18:16:52.172305 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:52.172272    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-787c864788-jdqjl" event={"ID":"e1728312-8558-4aec-86f6-c48c7dfdaffa","Type":"ContainerStarted","Data":"246ae6988ed9a39f9fa3a14e947998bdace23bdb080079f33838219d920fdf6c"}
Sep 11 18:16:52.172772 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:52.172320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-787c864788-jdqjl" event={"ID":"e1728312-8558-4aec-86f6-c48c7dfdaffa","Type":"ContainerStarted","Data":"c7e5d03791724c431e50fa18546ce379f1a9e61385b23a99e40d7780d4c091e7"}
Sep 11 18:16:52.172772 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:52.172358    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:16:52.191798 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:52.191755    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-787c864788-jdqjl" podStartSLOduration=17.191739363 podStartE2EDuration="17.191739363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:52.190884425 +0000 UTC m=+88.932011270" watchObservedRunningTime="2026-09-11 18:16:52.191739363 +0000 UTC m=+88.932866205"
Sep 11 18:16:56.787595 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:56.787550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:56.787595 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:56.787611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:56.788299 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:56.788274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96e20d2c-23a6-4e42-a459-e680cddc8572-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:56.789944 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:56.789924    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96e20d2c-23a6-4e42-a459-e680cddc8572-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bv5j5\" (UID: \"96e20d2c-23a6-4e42-a459-e680cddc8572\") " pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:56.902619 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:56.902591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bv5j5"
Sep 11 18:16:57.171928 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:57.171897    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bv5j5"]
Sep 11 18:16:57.175053 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:16:57.175025    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e20d2c_23a6_4e42_a459_e680cddc8572.slice/crio-04dd43815b9c96c7f9d78670b0ee944c399543e5b1b6f9ff89ac1acc53cd2391 WatchSource:0}: Error finding container 04dd43815b9c96c7f9d78670b0ee944c399543e5b1b6f9ff89ac1acc53cd2391: Status 404 returned error can't find the container with id 04dd43815b9c96c7f9d78670b0ee944c399543e5b1b6f9ff89ac1acc53cd2391
Sep 11 18:16:57.184273 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:57.184252    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bv5j5" event={"ID":"96e20d2c-23a6-4e42-a459-e680cddc8572","Type":"ContainerStarted","Data":"04dd43815b9c96c7f9d78670b0ee944c399543e5b1b6f9ff89ac1acc53cd2391"}
Sep 11 18:16:58.188855 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:58.188821    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bv5j5" event={"ID":"96e20d2c-23a6-4e42-a459-e680cddc8572","Type":"ContainerStarted","Data":"0a584c7305dbaed5429e7c8a7e6e633cae8b6cb3372aaa761abc2b28fc1f3c42"}
Sep 11 18:16:58.189189 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:16:58.188875    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bv5j5" event={"ID":"96e20d2c-23a6-4e42-a459-e680cddc8572","Type":"ContainerStarted","Data":"84c1df0b8d64a7a05bf75a04f05e052a408521586ac0590b6483b7b8d380b1b5"}
Sep 11 18:17:00.198416 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:00.198377    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bv5j5" event={"ID":"96e20d2c-23a6-4e42-a459-e680cddc8572","Type":"ContainerStarted","Data":"9b912f56682828cf1aa611a988394c0a561a857a95fe4062b7351cbc54090963"}
Sep 11 18:17:00.217823 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:00.217780    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bv5j5" podStartSLOduration=18.079184024 podStartE2EDuration="20.217767841s" totalImagesPullingTime="2.138583817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:16:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:00.216806339 +0000 UTC m=+96.957933183" watchObservedRunningTime="2026-09-11 18:17:00.217767841 +0000 UTC m=+96.958894684"
Sep 11 18:17:02.430729 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.430694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:17:02.433091 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.433066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e8a008ce-8cd5-4753-a3ad-7337832bf965-metrics-tls\") pod \"dns-default-lck25\" (UID: \"e8a008ce-8cd5-4753-a3ad-7337832bf965\") " pod="openshift-dns/dns-default-lck25"
Sep 11 18:17:02.492977 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.492948    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-b9ltz\""
Sep 11 18:17:02.501728 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.501712    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lck25"
Sep 11 18:17:02.594661 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.594628    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42"]
Sep 11 18:17:02.597664 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.597645    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42"
Sep 11 18:17:02.609730 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.609708    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 18:17:02.609824 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.609742    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 18:17:02.610671 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.610644    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 18:17:02.610671 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.610662    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 18:17:02.611662 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.611639    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-787c864788-jdqjl"]
Sep 11 18:17:02.612568 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.612550    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42"]
Sep 11 18:17:02.689017 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.688955    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-br52f"]
Sep 11 18:17:02.691667 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.691652    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-br52f"
Sep 11 18:17:02.693780 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.693749    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 11 18:17:02.693880 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.693796    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 11 18:17:02.693931 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.693886    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-s5cdz\""
Sep 11 18:17:02.703021 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.703001    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-br52f"]
Sep 11 18:17:02.732707 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.732683    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bptg\" (UniqueName: \"kubernetes.io/projected/f938aabd-fb5a-4b8f-afab-a37be4e4f443-kube-api-access-5bptg\") pod \"klusterlet-addon-workmgr-7fb9877fdc-z9z42\" (UID: \"f938aabd-fb5a-4b8f-afab-a37be4e4f443\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42"
Sep 11 18:17:02.732799 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.732712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f938aabd-fb5a-4b8f-afab-a37be4e4f443-tmp\") pod \"klusterlet-addon-workmgr-7fb9877fdc-z9z42\" (UID: \"f938aabd-fb5a-4b8f-afab-a37be4e4f443\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42"
Sep 11 18:17:02.732799 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.732736    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f938aabd-fb5a-4b8f-afab-a37be4e4f443-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fb9877fdc-z9z42\" (UID: \"f938aabd-fb5a-4b8f-afab-a37be4e4f443\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42"
Sep 11 18:17:02.769560 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.769541    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lck25"]
Sep 11 18:17:02.771947 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:17:02.771927    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a008ce_8cd5_4753_a3ad_7337832bf965.slice/crio-74410719fd3835ba967f0965b5088ceda47543eb0588645cbe2e26820f7035ea WatchSource:0}: Error finding container 74410719fd3835ba967f0965b5088ceda47543eb0588645cbe2e26820f7035ea: Status 404 returned error can't find the container with id 74410719fd3835ba967f0965b5088ceda47543eb0588645cbe2e26820f7035ea
Sep 11 18:17:02.833514 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.833468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bptg\" (UniqueName: \"kubernetes.io/projected/f938aabd-fb5a-4b8f-afab-a37be4e4f443-kube-api-access-5bptg\") pod \"klusterlet-addon-workmgr-7fb9877fdc-z9z42\" (UID: \"f938aabd-fb5a-4b8f-afab-a37be4e4f443\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42"
Sep 11 18:17:02.833645 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.833524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn5rp\" (UniqueName: \"kubernetes.io/projected/4e69db3f-e2e2-4726-80cc-3da6ca8c73ec-kube-api-access-kn5rp\") pod \"downloads-76f99584f4-br52f\" (UID: \"4e69db3f-e2e2-4726-80cc-3da6ca8c73ec\") " pod="openshift-console/downloads-76f99584f4-br52f"
Sep 11 18:17:02.833645 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.833559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f938aabd-fb5a-4b8f-afab-a37be4e4f443-tmp\") pod \"klusterlet-addon-workmgr-7fb9877fdc-z9z42\" (UID: \"f938aabd-fb5a-4b8f-afab-a37be4e4f443\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42"
Sep 11 18:17:02.833645 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.833587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f938aabd-fb5a-4b8f-afab-a37be4e4f443-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fb9877fdc-z9z42\" (UID: \"f938aabd-fb5a-4b8f-afab-a37be4e4f443\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42"
Sep 11 18:17:02.833998 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.833977    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f938aabd-fb5a-4b8f-afab-a37be4e4f443-tmp\") pod \"klusterlet-addon-workmgr-7fb9877fdc-z9z42\" (UID: \"f938aabd-fb5a-4b8f-afab-a37be4e4f443\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42"
Sep 11 18:17:02.836119 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.836102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f938aabd-fb5a-4b8f-afab-a37be4e4f443-klusterlet-config\") pod \"klusterlet-addon-workmgr-7fb9877fdc-z9z42\" (UID: \"f938aabd-fb5a-4b8f-afab-a37be4e4f443\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42"
Sep 11 18:17:02.850284 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.850267    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bptg\" (UniqueName: \"kubernetes.io/projected/f938aabd-fb5a-4b8f-afab-a37be4e4f443-kube-api-access-5bptg\") pod \"klusterlet-addon-workmgr-7fb9877fdc-z9z42\" (UID: \"f938aabd-fb5a-4b8f-afab-a37be4e4f443\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42"
Sep 11 18:17:02.906726 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.906702    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42"
Sep 11 18:17:02.934888 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.934852    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn5rp\" (UniqueName: \"kubernetes.io/projected/4e69db3f-e2e2-4726-80cc-3da6ca8c73ec-kube-api-access-kn5rp\") pod \"downloads-76f99584f4-br52f\" (UID: \"4e69db3f-e2e2-4726-80cc-3da6ca8c73ec\") " pod="openshift-console/downloads-76f99584f4-br52f"
Sep 11 18:17:02.967438 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.967410    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn5rp\" (UniqueName: \"kubernetes.io/projected/4e69db3f-e2e2-4726-80cc-3da6ca8c73ec-kube-api-access-kn5rp\") pod \"downloads-76f99584f4-br52f\" (UID: \"4e69db3f-e2e2-4726-80cc-3da6ca8c73ec\") " pod="openshift-console/downloads-76f99584f4-br52f"
Sep 11 18:17:02.999719 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:02.999697    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-br52f"
Sep 11 18:17:03.181313 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:03.181153    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42"]
Sep 11 18:17:03.183675 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:17:03.183650    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf938aabd_fb5a_4b8f_afab_a37be4e4f443.slice/crio-d9a4545316137d44764f2305bc3b23cd7f1672240e61554eadbf0366815718ea WatchSource:0}: Error finding container d9a4545316137d44764f2305bc3b23cd7f1672240e61554eadbf0366815718ea: Status 404 returned error can't find the container with id d9a4545316137d44764f2305bc3b23cd7f1672240e61554eadbf0366815718ea
Sep 11 18:17:03.208804 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:03.208750    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42" event={"ID":"f938aabd-fb5a-4b8f-afab-a37be4e4f443","Type":"ContainerStarted","Data":"d9a4545316137d44764f2305bc3b23cd7f1672240e61554eadbf0366815718ea"}
Sep 11 18:17:03.209840 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:03.209818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lck25" event={"ID":"e8a008ce-8cd5-4753-a3ad-7337832bf965","Type":"ContainerStarted","Data":"74410719fd3835ba967f0965b5088ceda47543eb0588645cbe2e26820f7035ea"}
Sep 11 18:17:03.268300 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:03.268277    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-br52f"]
Sep 11 18:17:03.271045 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:17:03.271024    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e69db3f_e2e2_4726_80cc_3da6ca8c73ec.slice/crio-e17bb8e7932c0a1dbdb13885f0152d09e4413a18e29fbb6a8d05d6a0affb038f WatchSource:0}: Error finding container e17bb8e7932c0a1dbdb13885f0152d09e4413a18e29fbb6a8d05d6a0affb038f: Status 404 returned error can't find the container with id e17bb8e7932c0a1dbdb13885f0152d09e4413a18e29fbb6a8d05d6a0affb038f
Sep 11 18:17:04.215679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:04.215639    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-br52f" event={"ID":"4e69db3f-e2e2-4726-80cc-3da6ca8c73ec","Type":"ContainerStarted","Data":"e17bb8e7932c0a1dbdb13885f0152d09e4413a18e29fbb6a8d05d6a0affb038f"}
Sep 11 18:17:06.134814 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:06.134777    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fxtlq"
Sep 11 18:17:06.226164 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:06.226127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lck25" event={"ID":"e8a008ce-8cd5-4753-a3ad-7337832bf965","Type":"ContainerStarted","Data":"cbb97c121d8f72cb0203ac88c68ac53e7718ef1d09ba854ac7d85b512ea8257f"}
Sep 11 18:17:06.226334 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:06.226195    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lck25" event={"ID":"e8a008ce-8cd5-4753-a3ad-7337832bf965","Type":"ContainerStarted","Data":"e083b156628c9eb712697026eefceb85e33d447edc1ae632c35554906f3b0cfd"}
Sep 11 18:17:06.226444 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:06.226398    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lck25"
Sep 11 18:17:07.066422 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:07.066384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:17:07.067047 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:07.067023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c230cd27-7b95-48a6-8cd8-9f10f8a87d8a-service-ca-bundle\") pod \"router-default-7dcc55bd66-526gs\" (UID: \"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a\") " pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:17:07.336412 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:07.336334    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:17:07.855792 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:07.855553    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lck25" podStartSLOduration=67.116395639 podStartE2EDuration="1m9.855528983s" totalImagesPullingTime="2.739133344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:06.245990954 +0000 UTC m=+102.987117815" watchObservedRunningTime="2026-09-11 18:17:07.855528983 +0000 UTC m=+104.596655823"
Sep 11 18:17:07.856955 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:07.856931    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7dcc55bd66-526gs"]
Sep 11 18:17:07.859933 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:17:07.859906    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc230cd27_7b95_48a6_8cd8_9f10f8a87d8a.slice/crio-f51f90ab621c9fc747be57d411e52dc6e05d2de196294d63c828faca1dc54ed8 WatchSource:0}: Error finding container f51f90ab621c9fc747be57d411e52dc6e05d2de196294d63c828faca1dc54ed8: Status 404 returned error can't find the container with id f51f90ab621c9fc747be57d411e52dc6e05d2de196294d63c828faca1dc54ed8
Sep 11 18:17:08.236358 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:08.236320    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42" event={"ID":"f938aabd-fb5a-4b8f-afab-a37be4e4f443","Type":"ContainerStarted","Data":"b802f4312b40ae46b48335987ef9a19af5d563f85c1118832e4b137a114ef65b"}
Sep 11 18:17:08.236613 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:08.236589    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42"
Sep 11 18:17:08.238025 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:08.238002    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42"
Sep 11 18:17:08.239047 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:08.239021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7dcc55bd66-526gs" event={"ID":"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a","Type":"ContainerStarted","Data":"83530fece4200d61b50ab269fe89eb8d889b61d7707f21b043c361957552b452"}
Sep 11 18:17:08.239143 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:08.239072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7dcc55bd66-526gs" event={"ID":"c230cd27-7b95-48a6-8cd8-9f10f8a87d8a","Type":"ContainerStarted","Data":"f51f90ab621c9fc747be57d411e52dc6e05d2de196294d63c828faca1dc54ed8"}
Sep 11 18:17:08.261666 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:08.261625    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7fb9877fdc-z9z42" podStartSLOduration=1.854641071 podStartE2EDuration="6.261613452s" totalImagesPullingTime="4.406972381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:08.260616606 +0000 UTC m=+105.001743451" watchObservedRunningTime="2026-09-11 18:17:08.261613452 +0000 UTC m=+105.002740285"
Sep 11 18:17:08.282470 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:08.282421    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7dcc55bd66-526gs" podStartSLOduration=33.282404647999996 podStartE2EDuration="33.282404648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:08.281162735 +0000 UTC m=+105.022289659" watchObservedRunningTime="2026-09-11 18:17:08.282404648 +0000 UTC m=+105.023531483"
Sep 11 18:17:08.336679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:08.336654    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:17:08.339599 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:08.339579    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:17:09.249002 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:09.248966    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:17:09.250265 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:09.250243    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7dcc55bd66-526gs"
Sep 11 18:17:10.412399 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:10.412355    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j2gmd"]
Sep 11 18:17:10.428856 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:10.428830    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j2gmd"]
Sep 11 18:17:10.428988 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:10.428936    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j2gmd"
Sep 11 18:17:10.430985 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:10.430961    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 11 18:17:10.431120 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:10.431014    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-wxlbz\""
Sep 11 18:17:10.595975 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:10.595943    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/705fef02-989f-44dd-b52b-9283d167e1f8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-j2gmd\" (UID: \"705fef02-989f-44dd-b52b-9283d167e1f8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j2gmd"
Sep 11 18:17:10.697037 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:10.696950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/705fef02-989f-44dd-b52b-9283d167e1f8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-j2gmd\" (UID: \"705fef02-989f-44dd-b52b-9283d167e1f8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j2gmd"
Sep 11 18:17:10.699861 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:10.699838    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/705fef02-989f-44dd-b52b-9283d167e1f8-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-j2gmd\" (UID: \"705fef02-989f-44dd-b52b-9283d167e1f8\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j2gmd"
Sep 11 18:17:10.739628 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:10.739419    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j2gmd"
Sep 11 18:17:11.017055 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:11.016827    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j2gmd"]
Sep 11 18:17:11.019825 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:17:11.019794    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705fef02_989f_44dd_b52b_9283d167e1f8.slice/crio-ba62400f55a6883a1050a258bf00277d4d8cd8def42488286818e57f92fa046b WatchSource:0}: Error finding container ba62400f55a6883a1050a258bf00277d4d8cd8def42488286818e57f92fa046b: Status 404 returned error can't find the container with id ba62400f55a6883a1050a258bf00277d4d8cd8def42488286818e57f92fa046b
Sep 11 18:17:11.256933 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:11.256902    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j2gmd" event={"ID":"705fef02-989f-44dd-b52b-9283d167e1f8","Type":"ContainerStarted","Data":"ba62400f55a6883a1050a258bf00277d4d8cd8def42488286818e57f92fa046b"}
Sep 11 18:17:12.619125 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:12.618999    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:17:13.266329 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.266293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j2gmd" event={"ID":"705fef02-989f-44dd-b52b-9283d167e1f8","Type":"ContainerStarted","Data":"8a36289db4f210959aaf62f5976cb2c8fe4007682cfae99451af16c90812e49a"}
Sep 11 18:17:13.266546 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.266498    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j2gmd"
Sep 11 18:17:13.272062 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.272037    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j2gmd"
Sep 11 18:17:13.282090 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.282049    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-j2gmd" podStartSLOduration=1.683626911 podStartE2EDuration="3.282034579s" totalImagesPullingTime="1.598407668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:13.280980455 +0000 UTC m=+110.022107298" watchObservedRunningTime="2026-09-11 18:17:13.282034579 +0000 UTC m=+110.023161421"
Sep 11 18:17:13.490551 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.490457    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"]
Sep 11 18:17:13.492768 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.492741    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"
Sep 11 18:17:13.495070 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.495045    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 18:17:13.495227 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.495107    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 11 18:17:13.495227 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.495045    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 18:17:13.495227 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.495049    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-sqvdv\""
Sep 11 18:17:13.495562 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.495542    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 11 18:17:13.495744 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.495730    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 18:17:13.505051 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.505032    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"]
Sep 11 18:17:13.619727 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.619641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac54b5e3-0538-4542-8848-97249c328418-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-hkj9k\" (UID: \"ac54b5e3-0538-4542-8848-97249c328418\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"
Sep 11 18:17:13.619727 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.619699    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nd2j\" (UniqueName: \"kubernetes.io/projected/ac54b5e3-0538-4542-8848-97249c328418-kube-api-access-7nd2j\") pod \"prometheus-operator-68bcd7f987-hkj9k\" (UID: \"ac54b5e3-0538-4542-8848-97249c328418\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"
Sep 11 18:17:13.620144 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.619737    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac54b5e3-0538-4542-8848-97249c328418-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-hkj9k\" (UID: \"ac54b5e3-0538-4542-8848-97249c328418\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"
Sep 11 18:17:13.620144 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.619835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac54b5e3-0538-4542-8848-97249c328418-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-hkj9k\" (UID: \"ac54b5e3-0538-4542-8848-97249c328418\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"
Sep 11 18:17:13.721203 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.721173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nd2j\" (UniqueName: \"kubernetes.io/projected/ac54b5e3-0538-4542-8848-97249c328418-kube-api-access-7nd2j\") pod \"prometheus-operator-68bcd7f987-hkj9k\" (UID: \"ac54b5e3-0538-4542-8848-97249c328418\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"
Sep 11 18:17:13.721363 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.721224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac54b5e3-0538-4542-8848-97249c328418-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-hkj9k\" (UID: \"ac54b5e3-0538-4542-8848-97249c328418\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"
Sep 11 18:17:13.721363 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.721280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac54b5e3-0538-4542-8848-97249c328418-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-hkj9k\" (UID: \"ac54b5e3-0538-4542-8848-97249c328418\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"
Sep 11 18:17:13.721363 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.721324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac54b5e3-0538-4542-8848-97249c328418-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-hkj9k\" (UID: \"ac54b5e3-0538-4542-8848-97249c328418\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"
Sep 11 18:17:13.721897 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:17:13.721438    2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 11 18:17:13.721897 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:17:13.721527    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac54b5e3-0538-4542-8848-97249c328418-prometheus-operator-tls podName:ac54b5e3-0538-4542-8848-97249c328418 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:14.221499447 +0000 UTC m=+110.962626278 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/ac54b5e3-0538-4542-8848-97249c328418-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-hkj9k" (UID: "ac54b5e3-0538-4542-8848-97249c328418") : secret "prometheus-operator-tls" not found
Sep 11 18:17:13.722063 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.722037    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac54b5e3-0538-4542-8848-97249c328418-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-hkj9k\" (UID: \"ac54b5e3-0538-4542-8848-97249c328418\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"
Sep 11 18:17:13.724112 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.724082    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ac54b5e3-0538-4542-8848-97249c328418-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-hkj9k\" (UID: \"ac54b5e3-0538-4542-8848-97249c328418\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"
Sep 11 18:17:13.730629 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:13.730610    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nd2j\" (UniqueName: \"kubernetes.io/projected/ac54b5e3-0538-4542-8848-97249c328418-kube-api-access-7nd2j\") pod \"prometheus-operator-68bcd7f987-hkj9k\" (UID: \"ac54b5e3-0538-4542-8848-97249c328418\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"
Sep 11 18:17:14.225800 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:14.225756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac54b5e3-0538-4542-8848-97249c328418-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-hkj9k\" (UID: \"ac54b5e3-0538-4542-8848-97249c328418\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"
Sep 11 18:17:14.228144 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:14.228115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ac54b5e3-0538-4542-8848-97249c328418-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-hkj9k\" (UID: \"ac54b5e3-0538-4542-8848-97249c328418\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"
Sep 11 18:17:14.403070 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:14.403037    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"
Sep 11 18:17:14.678622 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:14.678589    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k"]
Sep 11 18:17:14.681573 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:17:14.681549    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac54b5e3_0538_4542_8848_97249c328418.slice/crio-19f7fc6033524d22f9486b876a0846a291b823fe8c25f4ee6554a719b5f8d4c8 WatchSource:0}: Error finding container 19f7fc6033524d22f9486b876a0846a291b823fe8c25f4ee6554a719b5f8d4c8: Status 404 returned error can't find the container with id 19f7fc6033524d22f9486b876a0846a291b823fe8c25f4ee6554a719b5f8d4c8
Sep 11 18:17:15.275534 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:15.275498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k" event={"ID":"ac54b5e3-0538-4542-8848-97249c328418","Type":"ContainerStarted","Data":"19f7fc6033524d22f9486b876a0846a291b823fe8c25f4ee6554a719b5f8d4c8"}
Sep 11 18:17:16.233222 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.233194    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lck25"
Sep 11 18:17:16.560623 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.560544    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54f678d8fb-qnd9d"]
Sep 11 18:17:16.563358 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.563330    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.565674 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.565637    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 11 18:17:16.565792 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.565699    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 11 18:17:16.566336 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.566319    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 11 18:17:16.566417 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.566350    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 11 18:17:16.566466 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.566426    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 11 18:17:16.566793 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.566762    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nmqv8\""
Sep 11 18:17:16.574801 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.574778    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 11 18:17:16.576082 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.576061    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54f678d8fb-qnd9d"]
Sep 11 18:17:16.744018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.743984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-serving-cert\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.744018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.744026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-service-ca\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.744276 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.744131    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-trusted-ca-bundle\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.744276 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.744176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlpt4\" (UniqueName: \"kubernetes.io/projected/096a7cb7-ee6c-40e2-9939-a636e17ab151-kube-api-access-qlpt4\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.744276 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.744238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-oauth-config\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.744276 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.744271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-config\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.744518 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.744299    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-oauth-serving-cert\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.845024 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.844944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-serving-cert\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.845024 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.844989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-service-ca\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.845236 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.845159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-trusted-ca-bundle\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.845236 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.845203    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlpt4\" (UniqueName: \"kubernetes.io/projected/096a7cb7-ee6c-40e2-9939-a636e17ab151-kube-api-access-qlpt4\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.845348 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.845257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-oauth-config\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.845348 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.845292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-config\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.845348 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.845336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-oauth-serving-cert\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.845794 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.845769    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-service-ca\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.846041 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.846014    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-oauth-serving-cert\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.846226 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.846176    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-config\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.846226 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.846195    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-trusted-ca-bundle\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.848018 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.847992    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-serving-cert\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.848779 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.848757    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-oauth-config\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.854572 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.854552    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlpt4\" (UniqueName: \"kubernetes.io/projected/096a7cb7-ee6c-40e2-9939-a636e17ab151-kube-api-access-qlpt4\") pod \"console-54f678d8fb-qnd9d\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") " pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:16.876389 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:16.876366    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:22.108409 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:22.106451    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54f678d8fb-qnd9d"]
Sep 11 18:17:22.302224 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:22.302132    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f678d8fb-qnd9d" event={"ID":"096a7cb7-ee6c-40e2-9939-a636e17ab151","Type":"ContainerStarted","Data":"03a692c219c5140802902001d2e6e97d7abcc3af2455e556585939e91fc11633"}
Sep 11 18:17:22.304419 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:22.304361    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-br52f" event={"ID":"4e69db3f-e2e2-4726-80cc-3da6ca8c73ec","Type":"ContainerStarted","Data":"6876043bce3ff59f90192b397d23588b1e0cee44204d8f83a169f210f898b9fc"}
Sep 11 18:17:22.304868 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:22.304840    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-br52f"
Sep 11 18:17:22.314436 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:22.314413    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-br52f"
Sep 11 18:17:22.340095 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:22.339471    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-br52f" podStartSLOduration=1.7025981890000002 podStartE2EDuration="20.339447331s" totalImagesPullingTime="18.636849142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:22.322074315 +0000 UTC m=+119.063201158" watchObservedRunningTime="2026-09-11 18:17:22.339447331 +0000 UTC m=+119.080574174"
Sep 11 18:17:23.313026 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:23.312989    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k" event={"ID":"ac54b5e3-0538-4542-8848-97249c328418","Type":"ContainerStarted","Data":"a6ed902a67593ad1a89b79fa1ed9d7e45723010093362853fbe6c3cf30a12940"}
Sep 11 18:17:23.313582 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:23.313044    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k" event={"ID":"ac54b5e3-0538-4542-8848-97249c328418","Type":"ContainerStarted","Data":"bf314a21e24005eeb800df6507b38590269050136d911296bcdb819c7e6c3633"}
Sep 11 18:17:23.334954 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:23.334851    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-hkj9k" podStartSLOduration=1.939738793 podStartE2EDuration="10.334829037s" totalImagesPullingTime="8.395090244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:23.333758361 +0000 UTC m=+120.074885205" watchObservedRunningTime="2026-09-11 18:17:23.334829037 +0000 UTC m=+120.075955880"
Sep 11 18:17:24.838503 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:24.838443    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"]
Sep 11 18:17:24.866170 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:24.866139    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"]
Sep 11 18:17:24.866331 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:24.866197    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l7n9c"]
Sep 11 18:17:24.866790 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:24.866764    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"
Sep 11 18:17:24.868787 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:24.868762    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zhhjv\""
Sep 11 18:17:24.868902 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:24.868855    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 11 18:17:24.868966 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:24.868908    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 11 18:17:24.884433 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:24.884415    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:24.887060 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:24.886755    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 18:17:24.887060 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:24.886939    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 18:17:24.887384 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:24.887365    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9m256\""
Sep 11 18:17:24.887603 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:24.887586    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 18:17:25.010580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.010007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1079d587-e3dc-471f-8cdb-11d28d7ae5d1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-n7wxx\" (UID: \"1079d587-e3dc-471f-8cdb-11d28d7ae5d1\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"
Sep 11 18:17:25.010580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.010064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1079d587-e3dc-471f-8cdb-11d28d7ae5d1-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-n7wxx\" (UID: \"1079d587-e3dc-471f-8cdb-11d28d7ae5d1\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"
Sep 11 18:17:25.010580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.010095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2f7n\" (UniqueName: \"kubernetes.io/projected/c332143e-9818-402b-9020-b419b5ad7f74-kube-api-access-n2f7n\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.010580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.010132    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1079d587-e3dc-471f-8cdb-11d28d7ae5d1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-n7wxx\" (UID: \"1079d587-e3dc-471f-8cdb-11d28d7ae5d1\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"
Sep 11 18:17:25.010580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.010177    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c332143e-9818-402b-9020-b419b5ad7f74-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.010580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.010214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr76j\" (UniqueName: \"kubernetes.io/projected/1079d587-e3dc-471f-8cdb-11d28d7ae5d1-kube-api-access-zr76j\") pod \"openshift-state-metrics-67dcf8c979-n7wxx\" (UID: \"1079d587-e3dc-471f-8cdb-11d28d7ae5d1\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"
Sep 11 18:17:25.010580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.010240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c332143e-9818-402b-9020-b419b5ad7f74-node-exporter-textfile\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.010580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.010262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c332143e-9818-402b-9020-b419b5ad7f74-node-exporter-wtmp\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.010580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.010280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c332143e-9818-402b-9020-b419b5ad7f74-root\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.010580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.010302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c332143e-9818-402b-9020-b419b5ad7f74-metrics-client-ca\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.010580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.010347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c332143e-9818-402b-9020-b419b5ad7f74-sys\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.010580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.010369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c332143e-9818-402b-9020-b419b5ad7f74-node-exporter-tls\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.010580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.010394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c332143e-9818-402b-9020-b419b5ad7f74-node-exporter-accelerators-collector-config\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.110980 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.110908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c332143e-9818-402b-9020-b419b5ad7f74-sys\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.110980 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.110952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c332143e-9818-402b-9020-b419b5ad7f74-node-exporter-tls\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.111170 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.110981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c332143e-9818-402b-9020-b419b5ad7f74-node-exporter-accelerators-collector-config\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.111170 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.111013    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c332143e-9818-402b-9020-b419b5ad7f74-sys\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.111170 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.111017    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1079d587-e3dc-471f-8cdb-11d28d7ae5d1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-n7wxx\" (UID: \"1079d587-e3dc-471f-8cdb-11d28d7ae5d1\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"
Sep 11 18:17:25.111170 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.111067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1079d587-e3dc-471f-8cdb-11d28d7ae5d1-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-n7wxx\" (UID: \"1079d587-e3dc-471f-8cdb-11d28d7ae5d1\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"
Sep 11 18:17:25.111170 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.111096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2f7n\" (UniqueName: \"kubernetes.io/projected/c332143e-9818-402b-9020-b419b5ad7f74-kube-api-access-n2f7n\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.111170 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.111127    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1079d587-e3dc-471f-8cdb-11d28d7ae5d1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-n7wxx\" (UID: \"1079d587-e3dc-471f-8cdb-11d28d7ae5d1\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"
Sep 11 18:17:25.111170 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.111168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c332143e-9818-402b-9020-b419b5ad7f74-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.111451 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.111205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr76j\" (UniqueName: \"kubernetes.io/projected/1079d587-e3dc-471f-8cdb-11d28d7ae5d1-kube-api-access-zr76j\") pod \"openshift-state-metrics-67dcf8c979-n7wxx\" (UID: \"1079d587-e3dc-471f-8cdb-11d28d7ae5d1\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"
Sep 11 18:17:25.111451 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.111232    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c332143e-9818-402b-9020-b419b5ad7f74-node-exporter-textfile\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.111451 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.111264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c332143e-9818-402b-9020-b419b5ad7f74-node-exporter-wtmp\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.111451 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.111295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c332143e-9818-402b-9020-b419b5ad7f74-root\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.111451 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.111321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c332143e-9818-402b-9020-b419b5ad7f74-metrics-client-ca\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.112175 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.112138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c332143e-9818-402b-9020-b419b5ad7f74-metrics-client-ca\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.114428 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.112686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c332143e-9818-402b-9020-b419b5ad7f74-node-exporter-textfile\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.114428 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.112826    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c332143e-9818-402b-9020-b419b5ad7f74-node-exporter-wtmp\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.114428 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.112873    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c332143e-9818-402b-9020-b419b5ad7f74-root\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.114428 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.113740    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1079d587-e3dc-471f-8cdb-11d28d7ae5d1-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-n7wxx\" (UID: \"1079d587-e3dc-471f-8cdb-11d28d7ae5d1\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"
Sep 11 18:17:25.114428 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.114095    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1079d587-e3dc-471f-8cdb-11d28d7ae5d1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-n7wxx\" (UID: \"1079d587-e3dc-471f-8cdb-11d28d7ae5d1\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"
Sep 11 18:17:25.114428 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.114165    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c332143e-9818-402b-9020-b419b5ad7f74-node-exporter-accelerators-collector-config\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.116859 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.116839    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c332143e-9818-402b-9020-b419b5ad7f74-node-exporter-tls\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.118321 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.118275    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c332143e-9818-402b-9020-b419b5ad7f74-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.118871 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.118830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1079d587-e3dc-471f-8cdb-11d28d7ae5d1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-n7wxx\" (UID: \"1079d587-e3dc-471f-8cdb-11d28d7ae5d1\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"
Sep 11 18:17:25.122273 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.122249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr76j\" (UniqueName: \"kubernetes.io/projected/1079d587-e3dc-471f-8cdb-11d28d7ae5d1-kube-api-access-zr76j\") pod \"openshift-state-metrics-67dcf8c979-n7wxx\" (UID: \"1079d587-e3dc-471f-8cdb-11d28d7ae5d1\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"
Sep 11 18:17:25.123883 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.123802    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2f7n\" (UniqueName: \"kubernetes.io/projected/c332143e-9818-402b-9020-b419b5ad7f74-kube-api-access-n2f7n\") pod \"node-exporter-l7n9c\" (UID: \"c332143e-9818-402b-9020-b419b5ad7f74\") " pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.178958 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.178927    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"
Sep 11 18:17:25.197225 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.197162    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l7n9c"
Sep 11 18:17:25.751025 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:17:25.750988    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc332143e_9818_402b_9020_b419b5ad7f74.slice/crio-74e46d5c6261d76baba3cec48f598324dffa17076a04dffaf22fa452de6dde70 WatchSource:0}: Error finding container 74e46d5c6261d76baba3cec48f598324dffa17076a04dffaf22fa452de6dde70: Status 404 returned error can't find the container with id 74e46d5c6261d76baba3cec48f598324dffa17076a04dffaf22fa452de6dde70
Sep 11 18:17:25.954195 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.954163    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 18:17:25.974429 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.974399    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:25.975241 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.974758    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 18:17:25.976652 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.976634    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 11 18:17:25.976769 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.976681    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 11 18:17:25.976990 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.976914    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 11 18:17:25.976990 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.976925    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 11 18:17:25.977216 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.977199    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 11 18:17:25.977418 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.977401    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 11 18:17:25.977530 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.977511    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9lgtp\""
Sep 11 18:17:25.978060 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.977739    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 11 18:17:25.978060 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.977789    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 11 18:17:25.978060 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:25.977741    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 11 18:17:26.020400 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.020319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df23b4f0-1ab0-4450-96e5-aba957a0c89e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.020400 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.020353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-config-volume\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.020400 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.020388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.020703 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.020421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.020703 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.020503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df23b4f0-1ab0-4450-96e5-aba957a0c89e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.020703 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.020540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.020703 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.020566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df23b4f0-1ab0-4450-96e5-aba957a0c89e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.020703 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.020589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.021007 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.020710    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.021007 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.020746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nml5s\" (UniqueName: \"kubernetes.io/projected/df23b4f0-1ab0-4450-96e5-aba957a0c89e-kube-api-access-nml5s\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.021007 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.020787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-web-config\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.021007 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.020825    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df23b4f0-1ab0-4450-96e5-aba957a0c89e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.021007 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.020851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df23b4f0-1ab0-4450-96e5-aba957a0c89e-config-out\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.025407 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.025379    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx"]
Sep 11 18:17:26.075540 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:17:26.075163    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1079d587_e3dc_471f_8cdb_11d28d7ae5d1.slice/crio-008c32aff862e689657509cb0962553bead4457430755dcb246a334c182ea234 WatchSource:0}: Error finding container 008c32aff862e689657509cb0962553bead4457430755dcb246a334c182ea234: Status 404 returned error can't find the container with id 008c32aff862e689657509cb0962553bead4457430755dcb246a334c182ea234
Sep 11 18:17:26.121329 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.121301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.121447 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.121339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df23b4f0-1ab0-4450-96e5-aba957a0c89e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.121447 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.121355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.121447 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.121405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.121447 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.121434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nml5s\" (UniqueName: \"kubernetes.io/projected/df23b4f0-1ab0-4450-96e5-aba957a0c89e-kube-api-access-nml5s\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.121694 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.121456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-web-config\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.121694 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.121509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df23b4f0-1ab0-4450-96e5-aba957a0c89e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.121694 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.121541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df23b4f0-1ab0-4450-96e5-aba957a0c89e-config-out\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.121694 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.121597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df23b4f0-1ab0-4450-96e5-aba957a0c89e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.121694 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.121624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-config-volume\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.121694 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.121663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.121694 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.121691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.122020 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.121719    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df23b4f0-1ab0-4450-96e5-aba957a0c89e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.122020 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:17:26.121921    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/df23b4f0-1ab0-4450-96e5-aba957a0c89e-alertmanager-trusted-ca-bundle podName:df23b4f0-1ab0-4450-96e5-aba957a0c89e nodeName:}" failed. No retries permitted until 2026-09-11 18:17:26.621897144 +0000 UTC m=+123.363023981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/df23b4f0-1ab0-4450-96e5-aba957a0c89e-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "df23b4f0-1ab0-4450-96e5-aba957a0c89e") : configmap references non-existent config key: ca-bundle.crt
Sep 11 18:17:26.122020 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:17:26.121998    2581 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 11 18:17:26.122173 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:17:26.122050    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-secret-alertmanager-main-tls podName:df23b4f0-1ab0-4450-96e5-aba957a0c89e nodeName:}" failed. No retries permitted until 2026-09-11 18:17:26.622034404 +0000 UTC m=+123.363161231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "df23b4f0-1ab0-4450-96e5-aba957a0c89e") : secret "alertmanager-main-tls" not found
Sep 11 18:17:26.122173 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.122102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/df23b4f0-1ab0-4450-96e5-aba957a0c89e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.125397 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.125336    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df23b4f0-1ab0-4450-96e5-aba957a0c89e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.126625 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.126325    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.127178 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.127129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.127178 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.127166    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-config-volume\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.127875 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.127505    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-web-config\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.127875 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.127672    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/df23b4f0-1ab0-4450-96e5-aba957a0c89e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.127875 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.127828    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.128624 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.128601    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/df23b4f0-1ab0-4450-96e5-aba957a0c89e-config-out\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.129434 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.129413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.133924 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.133901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nml5s\" (UniqueName: \"kubernetes.io/projected/df23b4f0-1ab0-4450-96e5-aba957a0c89e-kube-api-access-nml5s\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.330334 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.330303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx" event={"ID":"1079d587-e3dc-471f-8cdb-11d28d7ae5d1","Type":"ContainerStarted","Data":"aa62c030a7737d4e17e6a9c7ea4d1e18e15306dddcd3ee636a02f16c2af222ef"}
Sep 11 18:17:26.330444 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.330387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx" event={"ID":"1079d587-e3dc-471f-8cdb-11d28d7ae5d1","Type":"ContainerStarted","Data":"008c32aff862e689657509cb0962553bead4457430755dcb246a334c182ea234"}
Sep 11 18:17:26.333555 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.333522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f678d8fb-qnd9d" event={"ID":"096a7cb7-ee6c-40e2-9939-a636e17ab151","Type":"ContainerStarted","Data":"1a7e6a2edf2bfcecdd28e1638724f1ba626f87ddf57fdba8d8f4e94f93d75229"}
Sep 11 18:17:26.335396 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.335370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l7n9c" event={"ID":"c332143e-9818-402b-9020-b419b5ad7f74","Type":"ContainerStarted","Data":"74e46d5c6261d76baba3cec48f598324dffa17076a04dffaf22fa452de6dde70"}
Sep 11 18:17:26.358588 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.357803    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54f678d8fb-qnd9d" podStartSLOduration=6.35775271 podStartE2EDuration="10.357788722s" totalImagesPullingTime="4.000036012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:26.357350701 +0000 UTC m=+123.098477544" watchObservedRunningTime="2026-09-11 18:17:26.357788722 +0000 UTC m=+123.098915565"
Sep 11 18:17:26.626264 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.626223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.626444 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.626327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df23b4f0-1ab0-4450-96e5-aba957a0c89e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.627398 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.627368    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df23b4f0-1ab0-4450-96e5-aba957a0c89e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.629503 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.629457    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/df23b4f0-1ab0-4450-96e5-aba957a0c89e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"df23b4f0-1ab0-4450-96e5-aba957a0c89e\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:26.877199 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.877119    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:26.877199 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.877190    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:26.882955 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.882931    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:26.887215 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:26.887190    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 11 18:17:27.236501 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:27.236395    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 11 18:17:27.306666 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:17:27.306634    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf23b4f0_1ab0_4450_96e5_aba957a0c89e.slice/crio-03c22b33906fd9538c9a22c7b9c079b608fa1e1db10e700cafddf6876c83b43f WatchSource:0}: Error finding container 03c22b33906fd9538c9a22c7b9c079b608fa1e1db10e700cafddf6876c83b43f: Status 404 returned error can't find the container with id 03c22b33906fd9538c9a22c7b9c079b608fa1e1db10e700cafddf6876c83b43f
Sep 11 18:17:27.342489 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:27.342386    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx" event={"ID":"1079d587-e3dc-471f-8cdb-11d28d7ae5d1","Type":"ContainerStarted","Data":"84b5343714e99609da106ef5d1ca17ac9224d42815537d310b1e0ad19f1c56b5"}
Sep 11 18:17:27.344111 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:27.344074    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df23b4f0-1ab0-4450-96e5-aba957a0c89e","Type":"ContainerStarted","Data":"03c22b33906fd9538c9a22c7b9c079b608fa1e1db10e700cafddf6876c83b43f"}
Sep 11 18:17:27.360092 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:27.360066    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:17:27.630240 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:27.630186    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-787c864788-jdqjl" podUID="e1728312-8558-4aec-86f6-c48c7dfdaffa" containerName="registry" containerID="cri-o://246ae6988ed9a39f9fa3a14e947998bdace23bdb080079f33838219d920fdf6c" gracePeriod=30
Sep 11 18:17:28.244590 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.244561    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:17:28.346337 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.346260    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1728312-8558-4aec-86f6-c48c7dfdaffa-installation-pull-secrets\") pod \"e1728312-8558-4aec-86f6-c48c7dfdaffa\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") "
Sep 11 18:17:28.346337 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.346310    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lfd5z\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-kube-api-access-lfd5z\") pod \"e1728312-8558-4aec-86f6-c48c7dfdaffa\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") "
Sep 11 18:17:28.346592 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.346344    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-bound-sa-token\") pod \"e1728312-8558-4aec-86f6-c48c7dfdaffa\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") "
Sep 11 18:17:28.346592 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.346397    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1728312-8558-4aec-86f6-c48c7dfdaffa-ca-trust-extracted\") pod \"e1728312-8558-4aec-86f6-c48c7dfdaffa\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") "
Sep 11 18:17:28.346592 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.346420    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls\") pod \"e1728312-8558-4aec-86f6-c48c7dfdaffa\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") "
Sep 11 18:17:28.346592 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.346453    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1728312-8558-4aec-86f6-c48c7dfdaffa-image-registry-private-configuration\") pod \"e1728312-8558-4aec-86f6-c48c7dfdaffa\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") "
Sep 11 18:17:28.346592 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.346521    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-certificates\") pod \"e1728312-8558-4aec-86f6-c48c7dfdaffa\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") "
Sep 11 18:17:28.346592 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.346566    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1728312-8558-4aec-86f6-c48c7dfdaffa-trusted-ca\") pod \"e1728312-8558-4aec-86f6-c48c7dfdaffa\" (UID: \"e1728312-8558-4aec-86f6-c48c7dfdaffa\") "
Sep 11 18:17:28.347155 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.347104    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1728312-8558-4aec-86f6-c48c7dfdaffa-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e1728312-8558-4aec-86f6-c48c7dfdaffa" (UID: "e1728312-8558-4aec-86f6-c48c7dfdaffa"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:17:28.347309 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.347226    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e1728312-8558-4aec-86f6-c48c7dfdaffa" (UID: "e1728312-8558-4aec-86f6-c48c7dfdaffa"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:17:28.349745 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.349704    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-kube-api-access-lfd5z" (OuterVolumeSpecName: "kube-api-access-lfd5z") pod "e1728312-8558-4aec-86f6-c48c7dfdaffa" (UID: "e1728312-8558-4aec-86f6-c48c7dfdaffa"). InnerVolumeSpecName "kube-api-access-lfd5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:17:28.350392 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.350348    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1728312-8558-4aec-86f6-c48c7dfdaffa-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e1728312-8558-4aec-86f6-c48c7dfdaffa" (UID: "e1728312-8558-4aec-86f6-c48c7dfdaffa"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:17:28.351267 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.351239    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1728312-8558-4aec-86f6-c48c7dfdaffa-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e1728312-8558-4aec-86f6-c48c7dfdaffa" (UID: "e1728312-8558-4aec-86f6-c48c7dfdaffa"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:17:28.353016 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.352567    2581 generic.go:356] "Generic (PLEG): container finished" podID="e1728312-8558-4aec-86f6-c48c7dfdaffa" containerID="246ae6988ed9a39f9fa3a14e947998bdace23bdb080079f33838219d920fdf6c" exitCode=0
Sep 11 18:17:28.353016 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.352618    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-787c864788-jdqjl"
Sep 11 18:17:28.353016 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.352646    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-787c864788-jdqjl" event={"ID":"e1728312-8558-4aec-86f6-c48c7dfdaffa","Type":"ContainerDied","Data":"246ae6988ed9a39f9fa3a14e947998bdace23bdb080079f33838219d920fdf6c"}
Sep 11 18:17:28.353016 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.352705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-787c864788-jdqjl" event={"ID":"e1728312-8558-4aec-86f6-c48c7dfdaffa","Type":"ContainerDied","Data":"c7e5d03791724c431e50fa18546ce379f1a9e61385b23a99e40d7780d4c091e7"}
Sep 11 18:17:28.354992 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.354848    2581 generic.go:356] "Generic (PLEG): container finished" podID="c332143e-9818-402b-9020-b419b5ad7f74" containerID="c95e5daff6dc775cd0f0df851a3ae05300f81a74d01f3de065cd4fa8a4504309" exitCode=0
Sep 11 18:17:28.354992 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.354905    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e1728312-8558-4aec-86f6-c48c7dfdaffa" (UID: "e1728312-8558-4aec-86f6-c48c7dfdaffa"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:17:28.354992 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.354983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l7n9c" event={"ID":"c332143e-9818-402b-9020-b419b5ad7f74","Type":"ContainerDied","Data":"c95e5daff6dc775cd0f0df851a3ae05300f81a74d01f3de065cd4fa8a4504309"}
Sep 11 18:17:28.355760 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.355718    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e1728312-8558-4aec-86f6-c48c7dfdaffa" (UID: "e1728312-8558-4aec-86f6-c48c7dfdaffa"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:17:28.357910 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.357886    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1728312-8558-4aec-86f6-c48c7dfdaffa-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e1728312-8558-4aec-86f6-c48c7dfdaffa" (UID: "e1728312-8558-4aec-86f6-c48c7dfdaffa"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:17:28.364558 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.364539    2581 scope.go:117] "RemoveContainer" containerID="246ae6988ed9a39f9fa3a14e947998bdace23bdb080079f33838219d920fdf6c"
Sep 11 18:17:28.447531 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.447499    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e1728312-8558-4aec-86f6-c48c7dfdaffa-ca-trust-extracted\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:17:28.447737 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.447720    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-tls\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:17:28.447844 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.447829    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e1728312-8558-4aec-86f6-c48c7dfdaffa-image-registry-private-configuration\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:17:28.447929 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.447916    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e1728312-8558-4aec-86f6-c48c7dfdaffa-registry-certificates\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:17:28.448016 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.448005    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e1728312-8558-4aec-86f6-c48c7dfdaffa-trusted-ca\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:17:28.448093 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.448082    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e1728312-8558-4aec-86f6-c48c7dfdaffa-installation-pull-secrets\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:17:28.450222 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.450195    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfd5z\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-kube-api-access-lfd5z\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:17:28.450330 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.450225    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1728312-8558-4aec-86f6-c48c7dfdaffa-bound-sa-token\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:17:28.678060 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.678008    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-787c864788-jdqjl"]
Sep 11 18:17:28.682683 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.682647    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-787c864788-jdqjl"]
Sep 11 18:17:28.795295 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.795260    2581 scope.go:117] "RemoveContainer" containerID="246ae6988ed9a39f9fa3a14e947998bdace23bdb080079f33838219d920fdf6c"
Sep 11 18:17:28.795727 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:17:28.795693    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246ae6988ed9a39f9fa3a14e947998bdace23bdb080079f33838219d920fdf6c\": container with ID starting with 246ae6988ed9a39f9fa3a14e947998bdace23bdb080079f33838219d920fdf6c not found: ID does not exist" containerID="246ae6988ed9a39f9fa3a14e947998bdace23bdb080079f33838219d920fdf6c"
Sep 11 18:17:28.795835 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:28.795738    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246ae6988ed9a39f9fa3a14e947998bdace23bdb080079f33838219d920fdf6c"} err="failed to get container status \"246ae6988ed9a39f9fa3a14e947998bdace23bdb080079f33838219d920fdf6c\": rpc error: code = NotFound desc = could not find container \"246ae6988ed9a39f9fa3a14e947998bdace23bdb080079f33838219d920fdf6c\": container with ID starting with 246ae6988ed9a39f9fa3a14e947998bdace23bdb080079f33838219d920fdf6c not found: ID does not exist"
Sep 11 18:17:29.182977 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.182950    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"]
Sep 11 18:17:29.183324 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.183306    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1728312-8558-4aec-86f6-c48c7dfdaffa" containerName="registry"
Sep 11 18:17:29.183402 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.183328    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1728312-8558-4aec-86f6-c48c7dfdaffa" containerName="registry"
Sep 11 18:17:29.183402 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.183395    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1728312-8558-4aec-86f6-c48c7dfdaffa" containerName="registry"
Sep 11 18:17:29.210492 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.209738    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"]
Sep 11 18:17:29.210492 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.209840    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.212194 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.212169    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ba2mseop9nm8b\""
Sep 11 18:17:29.212432 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.212413    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 11 18:17:29.212620 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.212596    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lc78w\""
Sep 11 18:17:29.212758 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.212739    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 11 18:17:29.212826 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.212816    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 11 18:17:29.212949 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.212931    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 11 18:17:29.258596 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.258573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64m5x\" (UniqueName: \"kubernetes.io/projected/57c68838-049b-41e9-957c-a00e44516563-kube-api-access-64m5x\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.258964 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.258671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/57c68838-049b-41e9-957c-a00e44516563-secret-metrics-server-tls\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.258964 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.258739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/57c68838-049b-41e9-957c-a00e44516563-client-ca-bundle\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.258964 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.258779    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c68838-049b-41e9-957c-a00e44516563-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.258964 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.258806    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/57c68838-049b-41e9-957c-a00e44516563-audit-log\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.258964 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.258859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/57c68838-049b-41e9-957c-a00e44516563-secret-metrics-server-client-certs\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.258964 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.258888    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/57c68838-049b-41e9-957c-a00e44516563-metrics-server-audit-profiles\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.359460 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.359427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64m5x\" (UniqueName: \"kubernetes.io/projected/57c68838-049b-41e9-957c-a00e44516563-kube-api-access-64m5x\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.359624 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.359522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/57c68838-049b-41e9-957c-a00e44516563-secret-metrics-server-tls\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.359624 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.359560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/57c68838-049b-41e9-957c-a00e44516563-client-ca-bundle\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.359624 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.359597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c68838-049b-41e9-957c-a00e44516563-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.359624 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.359620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/57c68838-049b-41e9-957c-a00e44516563-audit-log\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.359827 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.359664    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/57c68838-049b-41e9-957c-a00e44516563-secret-metrics-server-client-certs\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.359827 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.359694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/57c68838-049b-41e9-957c-a00e44516563-metrics-server-audit-profiles\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.360240 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.360213    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/57c68838-049b-41e9-957c-a00e44516563-audit-log\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.360750 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.360724    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/57c68838-049b-41e9-957c-a00e44516563-metrics-server-audit-profiles\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.360839 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.360824    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57c68838-049b-41e9-957c-a00e44516563-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.362873 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.362819    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/57c68838-049b-41e9-957c-a00e44516563-secret-metrics-server-tls\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.362873 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.362820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/57c68838-049b-41e9-957c-a00e44516563-client-ca-bundle\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.363258 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.363236    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/57c68838-049b-41e9-957c-a00e44516563-secret-metrics-server-client-certs\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.363343 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.363318    2581 generic.go:356] "Generic (PLEG): container finished" podID="df23b4f0-1ab0-4450-96e5-aba957a0c89e" containerID="51f6015d84382e1e3e4420f9a38f7ee9e0dffc6ddc30b627c61b241754370473" exitCode=0
Sep 11 18:17:29.363406 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.363387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df23b4f0-1ab0-4450-96e5-aba957a0c89e","Type":"ContainerDied","Data":"51f6015d84382e1e3e4420f9a38f7ee9e0dffc6ddc30b627c61b241754370473"}
Sep 11 18:17:29.365951 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.365927    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l7n9c" event={"ID":"c332143e-9818-402b-9020-b419b5ad7f74","Type":"ContainerStarted","Data":"772b5f67d50eb3f1c6e37e9809e57811a04a320329fec3792781670135665b04"}
Sep 11 18:17:29.366047 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.365978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l7n9c" event={"ID":"c332143e-9818-402b-9020-b419b5ad7f74","Type":"ContainerStarted","Data":"6f5d4afe9261f269b0eb4616f9cd7363796d3558562d88656fabbe5413742a0a"}
Sep 11 18:17:29.368427 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.368403    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx" event={"ID":"1079d587-e3dc-471f-8cdb-11d28d7ae5d1","Type":"ContainerStarted","Data":"bd30a408e047149db5367ab40e61c35771c0bbc16f64c3a314f93018504b39cb"}
Sep 11 18:17:29.368577 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.368553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64m5x\" (UniqueName: \"kubernetes.io/projected/57c68838-049b-41e9-957c-a00e44516563-kube-api-access-64m5x\") pod \"metrics-server-5b4fbbccdd-rtlqf\" (UID: \"57c68838-049b-41e9-957c-a00e44516563\") " pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.407798 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.407740    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l7n9c" podStartSLOduration=3.738142742 podStartE2EDuration="5.407720575s" totalImagesPullingTime="1.57630041s" totalInitContainerRuntime="93.277423ms" isStatefulPod=true podCreationTimestamp="2026-09-11 18:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:29.406680133 +0000 UTC m=+126.147807001" watchObservedRunningTime="2026-09-11 18:17:29.407720575 +0000 UTC m=+126.148847418"
Sep 11 18:17:29.426614 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.426518    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-n7wxx" podStartSLOduration=3.035157844 podStartE2EDuration="5.426499765s" totalImagesPullingTime="2.391341921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:29.425471746 +0000 UTC m=+126.166598589" watchObservedRunningTime="2026-09-11 18:17:29.426499765 +0000 UTC m=+126.167626609"
Sep 11 18:17:29.532937 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.532903    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:29.631507 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.630107    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-bm4mc"]
Sep 11 18:17:29.661823 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.661788    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-bm4mc"]
Sep 11 18:17:29.661983 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.661900    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bm4mc"
Sep 11 18:17:29.664129 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.664103    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 11 18:17:29.664242 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.664109    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-pcnrc\""
Sep 11 18:17:29.765304 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.764998    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/91949a49-5e56-4b47-bdaa-1900732d79e3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-bm4mc\" (UID: \"91949a49-5e56-4b47-bdaa-1900732d79e3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bm4mc"
Sep 11 18:17:29.765304 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.765084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvvqf\" (UniqueName: \"kubernetes.io/projected/91949a49-5e56-4b47-bdaa-1900732d79e3-kube-api-access-gvvqf\") pod \"monitoring-plugin-5d85778b9d-bm4mc\" (UID: \"91949a49-5e56-4b47-bdaa-1900732d79e3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bm4mc"
Sep 11 18:17:29.768144 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.767022    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1728312-8558-4aec-86f6-c48c7dfdaffa" path="/var/lib/kubelet/pods/e1728312-8558-4aec-86f6-c48c7dfdaffa/volumes"
Sep 11 18:17:29.853180 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.853146    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"]
Sep 11 18:17:29.857389 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:17:29.857348    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c68838_049b_41e9_957c_a00e44516563.slice/crio-9a4cbfa1fb585aaf5bd39ba66ecd11e9b7e88ce24d59abdfc5795e09879e9e53 WatchSource:0}: Error finding container 9a4cbfa1fb585aaf5bd39ba66ecd11e9b7e88ce24d59abdfc5795e09879e9e53: Status 404 returned error can't find the container with id 9a4cbfa1fb585aaf5bd39ba66ecd11e9b7e88ce24d59abdfc5795e09879e9e53
Sep 11 18:17:29.865630 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.865594    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvvqf\" (UniqueName: \"kubernetes.io/projected/91949a49-5e56-4b47-bdaa-1900732d79e3-kube-api-access-gvvqf\") pod \"monitoring-plugin-5d85778b9d-bm4mc\" (UID: \"91949a49-5e56-4b47-bdaa-1900732d79e3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bm4mc"
Sep 11 18:17:29.865734 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.865687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/91949a49-5e56-4b47-bdaa-1900732d79e3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-bm4mc\" (UID: \"91949a49-5e56-4b47-bdaa-1900732d79e3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bm4mc"
Sep 11 18:17:29.868599 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.868576    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/91949a49-5e56-4b47-bdaa-1900732d79e3-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-bm4mc\" (UID: \"91949a49-5e56-4b47-bdaa-1900732d79e3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bm4mc"
Sep 11 18:17:29.880318 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.880290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvvqf\" (UniqueName: \"kubernetes.io/projected/91949a49-5e56-4b47-bdaa-1900732d79e3-kube-api-access-gvvqf\") pod \"monitoring-plugin-5d85778b9d-bm4mc\" (UID: \"91949a49-5e56-4b47-bdaa-1900732d79e3\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bm4mc"
Sep 11 18:17:29.972956 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:29.972920    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bm4mc"
Sep 11 18:17:30.279896 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:30.279822    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-bm4mc"]
Sep 11 18:17:30.282822 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:17:30.282782    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91949a49_5e56_4b47_bdaa_1900732d79e3.slice/crio-7b16da0bdd68638ad1be8ca36f693fb41c33dc6aefd10e37affa0cbe00f28204 WatchSource:0}: Error finding container 7b16da0bdd68638ad1be8ca36f693fb41c33dc6aefd10e37affa0cbe00f28204: Status 404 returned error can't find the container with id 7b16da0bdd68638ad1be8ca36f693fb41c33dc6aefd10e37affa0cbe00f28204
Sep 11 18:17:30.375764 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:30.375642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bm4mc" event={"ID":"91949a49-5e56-4b47-bdaa-1900732d79e3","Type":"ContainerStarted","Data":"7b16da0bdd68638ad1be8ca36f693fb41c33dc6aefd10e37affa0cbe00f28204"}
Sep 11 18:17:30.378380 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:30.378350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf" event={"ID":"57c68838-049b-41e9-957c-a00e44516563","Type":"ContainerStarted","Data":"9a4cbfa1fb585aaf5bd39ba66ecd11e9b7e88ce24d59abdfc5795e09879e9e53"}
Sep 11 18:17:31.238908 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.235546    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:17:31.262350 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.262316    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:17:31.262549 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.262531    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.265835 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.265810    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 11 18:17:31.266091 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.266071    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 11 18:17:31.266091 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.266085    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 11 18:17:31.266227 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.266114    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 11 18:17:31.266443 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.266427    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fqpb5imuu8p8o\""
Sep 11 18:17:31.269507 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.267141    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 11 18:17:31.269507 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.267783    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 11 18:17:31.269507 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.268027    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 11 18:17:31.269507 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.268677    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 11 18:17:31.269507 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.268909    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 11 18:17:31.269507 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.269103    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 11 18:17:31.270074 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.270033    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 11 18:17:31.270621 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.270599    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lhs5r\""
Sep 11 18:17:31.270862 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.270843    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 11 18:17:31.271250 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.271231    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 11 18:17:31.274897 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.274878    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 11 18:17:31.279295 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279295 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-config\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279295 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279530 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279319    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be126077-8ac6-4292-a6fd-19c933373f63-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279530 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279346    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279530 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279530 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-web-config\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279530 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279418    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be126077-8ac6-4292-a6fd-19c933373f63-config-out\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279530 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9nx9\" (UniqueName: \"kubernetes.io/projected/be126077-8ac6-4292-a6fd-19c933373f63-kube-api-access-d9nx9\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279530 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279466    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279530 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/be126077-8ac6-4292-a6fd-19c933373f63-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279928 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279928 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279579    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279928 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279928 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279928 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279928 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279928 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279928 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.279928 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.279908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.380489 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.380660 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380505    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-config\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.380660 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.380660 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be126077-8ac6-4292-a6fd-19c933373f63-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.380660 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.380660 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.380953 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-web-config\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.380953 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be126077-8ac6-4292-a6fd-19c933373f63-config-out\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.380953 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9nx9\" (UniqueName: \"kubernetes.io/projected/be126077-8ac6-4292-a6fd-19c933373f63-kube-api-access-d9nx9\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.380953 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.380953 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/be126077-8ac6-4292-a6fd-19c933373f63-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.380953 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.380953 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.380953 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380891    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.380953 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.380953 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.381529 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.380972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.381529 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.381030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.381529 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.381057    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.381529 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.381089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.382382 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.382350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.383206 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.383174    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/be126077-8ac6-4292-a6fd-19c933373f63-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.383410 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.383383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.384102 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.384073    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.385311 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.384988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.385530 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.385505    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.385619 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.385517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.386202 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.386179    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.386564 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.386537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.386695 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.386673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/be126077-8ac6-4292-a6fd-19c933373f63-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.387804 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.387779    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.388739 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.388713    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.390539 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.390354    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.390539 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.390411    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.391107 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.390929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be126077-8ac6-4292-a6fd-19c933373f63-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.391107 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.391072    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.391461 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.391420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be126077-8ac6-4292-a6fd-19c933373f63-config-out\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.391990 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.391949    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-config\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.392454 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.392415    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be126077-8ac6-4292-a6fd-19c933373f63-web-config\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.396434 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.396409    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9nx9\" (UniqueName: \"kubernetes.io/projected/be126077-8ac6-4292-a6fd-19c933373f63-kube-api-access-d9nx9\") pod \"prometheus-k8s-0\" (UID: \"be126077-8ac6-4292-a6fd-19c933373f63\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:31.579536 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:31.579440    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:33.359867 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:33.359838    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 11 18:17:33.362825 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:17:33.362795    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe126077_8ac6_4292_a6fd_19c933373f63.slice/crio-27a3a93f6fdd01a5f64988855cf90fb86bdd0a80d1bad4806a8e719379b0caa0 WatchSource:0}: Error finding container 27a3a93f6fdd01a5f64988855cf90fb86bdd0a80d1bad4806a8e719379b0caa0: Status 404 returned error can't find the container with id 27a3a93f6fdd01a5f64988855cf90fb86bdd0a80d1bad4806a8e719379b0caa0
Sep 11 18:17:33.396395 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:33.396373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bm4mc" event={"ID":"91949a49-5e56-4b47-bdaa-1900732d79e3","Type":"ContainerStarted","Data":"b72951b28df44976a8188ab27db5e7212be9f516876e7b1917974e0f99438599"}
Sep 11 18:17:33.397829 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:33.397801    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be126077-8ac6-4292-a6fd-19c933373f63","Type":"ContainerStarted","Data":"27a3a93f6fdd01a5f64988855cf90fb86bdd0a80d1bad4806a8e719379b0caa0"}
Sep 11 18:17:33.399454 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:33.399410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf" event={"ID":"57c68838-049b-41e9-957c-a00e44516563","Type":"ContainerStarted","Data":"7e88f59b3d1e3bc835874a688054b32854a7a96d4280dc4afd05709d5a78eb8a"}
Sep 11 18:17:33.401858 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:33.401831    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df23b4f0-1ab0-4450-96e5-aba957a0c89e","Type":"ContainerStarted","Data":"c48ccdc0cae02c5f43981a341b91ced6dac6c49c4fdeeab28576a8b157ebaedd"}
Sep 11 18:17:33.401959 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:33.401903    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df23b4f0-1ab0-4450-96e5-aba957a0c89e","Type":"ContainerStarted","Data":"6e9978ee0de464bcbae0d73c4a6f2349db9496e436f85a95e5d9b45ae242c1ac"}
Sep 11 18:17:33.417205 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:33.416913    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bm4mc"
Sep 11 18:17:33.422899 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:33.422863    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bm4mc"
Sep 11 18:17:33.433294 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:33.433253    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-bm4mc" podStartSLOduration=1.630202978 podStartE2EDuration="4.433239549s" totalImagesPullingTime="2.803036571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:33.4315859 +0000 UTC m=+130.172712742" watchObservedRunningTime="2026-09-11 18:17:33.433239549 +0000 UTC m=+130.174366389"
Sep 11 18:17:33.449565 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:33.449529    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf" podStartSLOduration=1.224136456 podStartE2EDuration="4.449517008s" totalImagesPullingTime="3.225380552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:33.448311056 +0000 UTC m=+130.189437922" watchObservedRunningTime="2026-09-11 18:17:33.449517008 +0000 UTC m=+130.190643868"
Sep 11 18:17:33.503357 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:33.503317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs\") pod \"network-metrics-daemon-48sl5\" (UID: \"db03c3a4-cfd5-443f-afe5-6e2fad4b0678\") " pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:17:33.505880 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:33.505856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db03c3a4-cfd5-443f-afe5-6e2fad4b0678-metrics-certs\") pod \"network-metrics-daemon-48sl5\" (UID: \"db03c3a4-cfd5-443f-afe5-6e2fad4b0678\") " pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:17:33.569309 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:33.569284    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-28rql\""
Sep 11 18:17:33.577805 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:33.577783    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48sl5"
Sep 11 18:17:33.856048 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:33.855712    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-48sl5"]
Sep 11 18:17:33.858156 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:17:33.858124    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb03c3a4_cfd5_443f_afe5_6e2fad4b0678.slice/crio-ff994f1b57fe0e54cd6d3be0bd1e47a157af7ab554ae7cc8eeb16c1d7283cb25 WatchSource:0}: Error finding container ff994f1b57fe0e54cd6d3be0bd1e47a157af7ab554ae7cc8eeb16c1d7283cb25: Status 404 returned error can't find the container with id ff994f1b57fe0e54cd6d3be0bd1e47a157af7ab554ae7cc8eeb16c1d7283cb25
Sep 11 18:17:34.407930 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:34.407898    2581 generic.go:356] "Generic (PLEG): container finished" podID="be126077-8ac6-4292-a6fd-19c933373f63" containerID="0fc092885b4208cc9aa4fc5eeaadfbb8cb15d5c0a3f039cd859923a9a1d9c9b2" exitCode=0
Sep 11 18:17:34.408494 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:34.407991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be126077-8ac6-4292-a6fd-19c933373f63","Type":"ContainerDied","Data":"0fc092885b4208cc9aa4fc5eeaadfbb8cb15d5c0a3f039cd859923a9a1d9c9b2"}
Sep 11 18:17:34.409220 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:34.409190    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-48sl5" event={"ID":"db03c3a4-cfd5-443f-afe5-6e2fad4b0678","Type":"ContainerStarted","Data":"ff994f1b57fe0e54cd6d3be0bd1e47a157af7ab554ae7cc8eeb16c1d7283cb25"}
Sep 11 18:17:34.412446 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:34.412421    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df23b4f0-1ab0-4450-96e5-aba957a0c89e","Type":"ContainerStarted","Data":"0a033edb6f6ff4c111de083fe5525759b15b629a8e1102667b826c8a7e02c575"}
Sep 11 18:17:34.412561 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:34.412512    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df23b4f0-1ab0-4450-96e5-aba957a0c89e","Type":"ContainerStarted","Data":"17c81552628d891f6650d0b578ef5018b4806a175ed5ed0321636b60b1808121"}
Sep 11 18:17:34.412561 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:34.412556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df23b4f0-1ab0-4450-96e5-aba957a0c89e","Type":"ContainerStarted","Data":"d936f39fe3b4a82ccd539827ef7af10cb6d3f7e860b5ea2ae7359c7abdf7a19a"}
Sep 11 18:17:36.421898 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:36.421862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-48sl5" event={"ID":"db03c3a4-cfd5-443f-afe5-6e2fad4b0678","Type":"ContainerStarted","Data":"44615590e9fb3d8e3b50e0b915635050e876791a2ca227352a0005fc88c68af8"}
Sep 11 18:17:36.422431 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:36.421911    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-48sl5" event={"ID":"db03c3a4-cfd5-443f-afe5-6e2fad4b0678","Type":"ContainerStarted","Data":"b88aeea8adc81929351420bb26f0c0ea89c4f8d1e6d5d4a6f86363c06fc5130f"}
Sep 11 18:17:36.425225 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:36.425202    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"df23b4f0-1ab0-4450-96e5-aba957a0c89e","Type":"ContainerStarted","Data":"29d9d33405cd8a649354168c2cb7648a98c82b7f65192ef02d512aa0720a286d"}
Sep 11 18:17:36.438438 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:36.438394    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-48sl5" podStartSLOduration=130.708699731 podStartE2EDuration="2m12.438378027s" totalImagesPullingTime="1.729678296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:36.43802152 +0000 UTC m=+133.179148363" watchObservedRunningTime="2026-09-11 18:17:36.438378027 +0000 UTC m=+133.179504871"
Sep 11 18:17:36.467722 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:36.467663    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.117790078 podStartE2EDuration="11.467642976s" totalImagesPullingTime="7.328447548s" totalInitContainerRuntime="21.40535ms" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:36.465244224 +0000 UTC m=+133.206371097" watchObservedRunningTime="2026-09-11 18:17:36.467642976 +0000 UTC m=+133.208769819"
Sep 11 18:17:39.440177 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:39.440143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be126077-8ac6-4292-a6fd-19c933373f63","Type":"ContainerStarted","Data":"7df9d5b5aaed4e179853baa6bea173e5bdb9a9be4473ba3552789d98609cbd8b"}
Sep 11 18:17:39.440643 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:39.440230    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be126077-8ac6-4292-a6fd-19c933373f63","Type":"ContainerStarted","Data":"b452a54a3a72080f78b8c5f51edafc22653040e84c716cae29696927a7cc0ad3"}
Sep 11 18:17:42.455927 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:42.455887    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be126077-8ac6-4292-a6fd-19c933373f63","Type":"ContainerStarted","Data":"c3dc69ed4a04ae4bba476b43fddb0e6b5a3f2b206fc059e742b57382a86b42be"}
Sep 11 18:17:42.456459 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:42.455996    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be126077-8ac6-4292-a6fd-19c933373f63","Type":"ContainerStarted","Data":"0d3c4bd42b27805b3c2ae756eb908b58723a4276641a79d14ae5a25d856c3cf0"}
Sep 11 18:17:42.456459 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:42.456042    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be126077-8ac6-4292-a6fd-19c933373f63","Type":"ContainerStarted","Data":"b7864a756e56b4a2142b777c4d45eb779f48d10c723d361cc21ed0cd2a137831"}
Sep 11 18:17:42.456459 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:42.456080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"be126077-8ac6-4292-a6fd-19c933373f63","Type":"ContainerStarted","Data":"4e8dc1528dfcdf8df68a0456634aecba410dae91d0c1497130e93496532d7008"}
Sep 11 18:17:42.486716 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:42.486669    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.584128378 podStartE2EDuration="11.486647014s" totalImagesPullingTime="6.868412429s" totalInitContainerRuntime="34.106207ms" isStatefulPod=false podCreationTimestamp="2026-09-11 18:17:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:17:42.484838068 +0000 UTC m=+139.225964911" watchObservedRunningTime="2026-09-11 18:17:42.486647014 +0000 UTC m=+139.227773872"
Sep 11 18:17:44.811384 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:44.811349    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54f678d8fb-qnd9d"]
Sep 11 18:17:46.580195 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:46.580161    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:17:49.533726 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:49.533694    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:17:49.534094 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:17:49.533748    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:18:09.538270 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:09.538239    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:18:09.541999 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:09.541971    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5b4fbbccdd-rtlqf"
Sep 11 18:18:09.830730 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:09.830640    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-54f678d8fb-qnd9d" podUID="096a7cb7-ee6c-40e2-9939-a636e17ab151" containerName="console" containerID="cri-o://1a7e6a2edf2bfcecdd28e1638724f1ba626f87ddf57fdba8d8f4e94f93d75229" gracePeriod=15
Sep 11 18:18:10.104121 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.104067    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:18:10.209930 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.209899    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-trusted-ca-bundle\") pod \"096a7cb7-ee6c-40e2-9939-a636e17ab151\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") "
Sep 11 18:18:10.210098 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.209962    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-config\") pod \"096a7cb7-ee6c-40e2-9939-a636e17ab151\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") "
Sep 11 18:18:10.210098 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.210006    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-serving-cert\") pod \"096a7cb7-ee6c-40e2-9939-a636e17ab151\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") "
Sep 11 18:18:10.210098 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.210042    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlpt4\" (UniqueName: \"kubernetes.io/projected/096a7cb7-ee6c-40e2-9939-a636e17ab151-kube-api-access-qlpt4\") pod \"096a7cb7-ee6c-40e2-9939-a636e17ab151\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") "
Sep 11 18:18:10.210098 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.210088    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-oauth-config\") pod \"096a7cb7-ee6c-40e2-9939-a636e17ab151\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") "
Sep 11 18:18:10.210327 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.210130    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-service-ca\") pod \"096a7cb7-ee6c-40e2-9939-a636e17ab151\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") "
Sep 11 18:18:10.210327 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.210155    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-oauth-serving-cert\") pod \"096a7cb7-ee6c-40e2-9939-a636e17ab151\" (UID: \"096a7cb7-ee6c-40e2-9939-a636e17ab151\") "
Sep 11 18:18:10.210327 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.210278    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "096a7cb7-ee6c-40e2-9939-a636e17ab151" (UID: "096a7cb7-ee6c-40e2-9939-a636e17ab151"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:18:10.210495 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.210432    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-trusted-ca-bundle\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:18:10.210645 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.210617    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "096a7cb7-ee6c-40e2-9939-a636e17ab151" (UID: "096a7cb7-ee6c-40e2-9939-a636e17ab151"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:18:10.210713 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.210663    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-service-ca" (OuterVolumeSpecName: "service-ca") pod "096a7cb7-ee6c-40e2-9939-a636e17ab151" (UID: "096a7cb7-ee6c-40e2-9939-a636e17ab151"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:18:10.210800 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.210719    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-config" (OuterVolumeSpecName: "console-config") pod "096a7cb7-ee6c-40e2-9939-a636e17ab151" (UID: "096a7cb7-ee6c-40e2-9939-a636e17ab151"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:18:10.212753 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.212731    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "096a7cb7-ee6c-40e2-9939-a636e17ab151" (UID: "096a7cb7-ee6c-40e2-9939-a636e17ab151"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:18:10.212954 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.212928    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096a7cb7-ee6c-40e2-9939-a636e17ab151-kube-api-access-qlpt4" (OuterVolumeSpecName: "kube-api-access-qlpt4") pod "096a7cb7-ee6c-40e2-9939-a636e17ab151" (UID: "096a7cb7-ee6c-40e2-9939-a636e17ab151"). InnerVolumeSpecName "kube-api-access-qlpt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:18:10.213038 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.213020    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "096a7cb7-ee6c-40e2-9939-a636e17ab151" (UID: "096a7cb7-ee6c-40e2-9939-a636e17ab151"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:18:10.311059 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.311027    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-oauth-config\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:18:10.311059 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.311053    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-service-ca\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:18:10.311059 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.311062    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-oauth-serving-cert\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:18:10.311271 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.311070    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-config\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:18:10.311271 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.311079    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/096a7cb7-ee6c-40e2-9939-a636e17ab151-console-serving-cert\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:18:10.311271 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.311087    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlpt4\" (UniqueName: \"kubernetes.io/projected/096a7cb7-ee6c-40e2-9939-a636e17ab151-kube-api-access-qlpt4\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:18:10.573805 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.573768    2581 generic.go:356] "Generic (PLEG): container finished" podID="096a7cb7-ee6c-40e2-9939-a636e17ab151" containerID="1a7e6a2edf2bfcecdd28e1638724f1ba626f87ddf57fdba8d8f4e94f93d75229" exitCode=0
Sep 11 18:18:10.574198 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.573815    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54f678d8fb-qnd9d"
Sep 11 18:18:10.574198 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.573834    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f678d8fb-qnd9d" event={"ID":"096a7cb7-ee6c-40e2-9939-a636e17ab151","Type":"ContainerDied","Data":"1a7e6a2edf2bfcecdd28e1638724f1ba626f87ddf57fdba8d8f4e94f93d75229"}
Sep 11 18:18:10.574198 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.573880    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54f678d8fb-qnd9d" event={"ID":"096a7cb7-ee6c-40e2-9939-a636e17ab151","Type":"ContainerDied","Data":"03a692c219c5140802902001d2e6e97d7abcc3af2455e556585939e91fc11633"}
Sep 11 18:18:10.574198 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.573901    2581 scope.go:117] "RemoveContainer" containerID="1a7e6a2edf2bfcecdd28e1638724f1ba626f87ddf57fdba8d8f4e94f93d75229"
Sep 11 18:18:10.582231 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.582216    2581 scope.go:117] "RemoveContainer" containerID="1a7e6a2edf2bfcecdd28e1638724f1ba626f87ddf57fdba8d8f4e94f93d75229"
Sep 11 18:18:10.582491 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:18:10.582455    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a7e6a2edf2bfcecdd28e1638724f1ba626f87ddf57fdba8d8f4e94f93d75229\": container with ID starting with 1a7e6a2edf2bfcecdd28e1638724f1ba626f87ddf57fdba8d8f4e94f93d75229 not found: ID does not exist" containerID="1a7e6a2edf2bfcecdd28e1638724f1ba626f87ddf57fdba8d8f4e94f93d75229"
Sep 11 18:18:10.582555 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.582523    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a7e6a2edf2bfcecdd28e1638724f1ba626f87ddf57fdba8d8f4e94f93d75229"} err="failed to get container status \"1a7e6a2edf2bfcecdd28e1638724f1ba626f87ddf57fdba8d8f4e94f93d75229\": rpc error: code = NotFound desc = could not find container \"1a7e6a2edf2bfcecdd28e1638724f1ba626f87ddf57fdba8d8f4e94f93d75229\": container with ID starting with 1a7e6a2edf2bfcecdd28e1638724f1ba626f87ddf57fdba8d8f4e94f93d75229 not found: ID does not exist"
Sep 11 18:18:10.595700 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.595677    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54f678d8fb-qnd9d"]
Sep 11 18:18:10.600609 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:10.600591    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54f678d8fb-qnd9d"]
Sep 11 18:18:11.764532 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:11.764457    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="096a7cb7-ee6c-40e2-9939-a636e17ab151" path="/var/lib/kubelet/pods/096a7cb7-ee6c-40e2-9939-a636e17ab151/volumes"
Sep 11 18:18:31.579849 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:31.579811    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:31.596039 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:31.596011    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:31.681367 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:31.681341    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 11 18:18:49.284591 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.284560    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-54d8895f84-wkfkb"]
Sep 11 18:18:49.284963 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.284852    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="096a7cb7-ee6c-40e2-9939-a636e17ab151" containerName="console"
Sep 11 18:18:49.284963 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.284862    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="096a7cb7-ee6c-40e2-9939-a636e17ab151" containerName="console"
Sep 11 18:18:49.284963 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.284924    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="096a7cb7-ee6c-40e2-9939-a636e17ab151" containerName="console"
Sep 11 18:18:49.287187 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.287166    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.289793 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.289768    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 11 18:18:49.289914 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.289815    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 11 18:18:49.289914 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.289859    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-64vwc\""
Sep 11 18:18:49.289914 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.289884    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 11 18:18:49.290075 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.289978    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 11 18:18:49.290216 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.290201    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 11 18:18:49.296392 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.296364    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 11 18:18:49.302897 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.302878    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54d8895f84-wkfkb"]
Sep 11 18:18:49.417144 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.417112    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.417144 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.417144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-metrics-client-ca\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.417365 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.417168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.417365 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.417214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt9mt\" (UniqueName: \"kubernetes.io/projected/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-kube-api-access-vt9mt\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.417365 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.417252    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-federate-client-tls\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.417365 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.417274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-telemeter-client-tls\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.417365 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.417314    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-serving-certs-ca-bundle\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.417365 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.417337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-secret-telemeter-client\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.517864 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.517833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-telemeter-client-tls\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.517864 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.517868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-serving-certs-ca-bundle\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.518072 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.517894    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-secret-telemeter-client\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.518072 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.517939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.518072 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.517966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-metrics-client-ca\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.518072 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.517997    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.518072 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.518030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt9mt\" (UniqueName: \"kubernetes.io/projected/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-kube-api-access-vt9mt\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.518293 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.518086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-federate-client-tls\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.518683 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.518648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-serving-certs-ca-bundle\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.518683 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.518702    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-metrics-client-ca\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.519177 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.519153    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-telemeter-trusted-ca-bundle\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.520698 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.520669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.520799 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.520750    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-federate-client-tls\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.520859 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.520818    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-secret-telemeter-client\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.520859 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.520823    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-telemeter-client-tls\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.526895 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.526874    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt9mt\" (UniqueName: \"kubernetes.io/projected/0dc77db2-d8b5-49f3-8894-f5a193c9fb08-kube-api-access-vt9mt\") pod \"telemeter-client-54d8895f84-wkfkb\" (UID: \"0dc77db2-d8b5-49f3-8894-f5a193c9fb08\") " pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.598520 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.598438    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb"
Sep 11 18:18:49.874749 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:49.874714    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-54d8895f84-wkfkb"]
Sep 11 18:18:49.875862 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:18:49.875834    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc77db2_d8b5_49f3_8894_f5a193c9fb08.slice/crio-619fa2b4e81ef92ede2d99d307ee0bd981b0ab1197ccb566919a670f20bd8d09 WatchSource:0}: Error finding container 619fa2b4e81ef92ede2d99d307ee0bd981b0ab1197ccb566919a670f20bd8d09: Status 404 returned error can't find the container with id 619fa2b4e81ef92ede2d99d307ee0bd981b0ab1197ccb566919a670f20bd8d09
Sep 11 18:18:50.744428 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:50.744388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb" event={"ID":"0dc77db2-d8b5-49f3-8894-f5a193c9fb08","Type":"ContainerStarted","Data":"619fa2b4e81ef92ede2d99d307ee0bd981b0ab1197ccb566919a670f20bd8d09"}
Sep 11 18:18:51.751569 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:51.751533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb" event={"ID":"0dc77db2-d8b5-49f3-8894-f5a193c9fb08","Type":"ContainerStarted","Data":"28703b63f9c8c80c14b8f1650e72dc8c78e0dd6dfc1339a595639e2b914acbbb"}
Sep 11 18:18:51.751927 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:51.751594    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb" event={"ID":"0dc77db2-d8b5-49f3-8894-f5a193c9fb08","Type":"ContainerStarted","Data":"5b2387c99eb56be2729a8d170ad910c833846065cf15f48acc82ec1097db5059"}
Sep 11 18:18:51.751927 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:51.751614    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb" event={"ID":"0dc77db2-d8b5-49f3-8894-f5a193c9fb08","Type":"ContainerStarted","Data":"6c9d58b86c5f061d79846e3e4eb35eba346f1f19d33cd35af823054ebae0ed95"}
Sep 11 18:18:51.775635 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:18:51.775586    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-54d8895f84-wkfkb" podStartSLOduration=1.281072714 podStartE2EDuration="2.775568444s" totalImagesPullingTime="1.49449573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:51.77340163 +0000 UTC m=+208.514528472" watchObservedRunningTime="2026-09-11 18:18:51.775568444 +0000 UTC m=+208.516695287"
Sep 11 18:20:23.624352 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:20:23.624321    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 18:21:17.824849 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:17.824772    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"]
Sep 11 18:21:17.831311 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:17.831281    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"
Sep 11 18:21:17.833612 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:17.833587    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 11 18:21:17.834537 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:17.834516    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-kq4q9\""
Sep 11 18:21:17.834654 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:17.834570    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 11 18:21:17.834654 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:17.834633    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 11 18:21:17.834763 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:17.834669    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:21:17.834763 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:17.834635    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 11 18:21:17.839436 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:17.839412    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"]
Sep 11 18:21:17.910261 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:17.910229    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ed83167-16c8-417b-9f27-077eefaf338b-metrics-cert\") pod \"lws-controller-manager-5588b8499b-6r6r7\" (UID: \"0ed83167-16c8-417b-9f27-077eefaf338b\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"
Sep 11 18:21:17.910420 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:17.910296    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4d2v\" (UniqueName: \"kubernetes.io/projected/0ed83167-16c8-417b-9f27-077eefaf338b-kube-api-access-z4d2v\") pod \"lws-controller-manager-5588b8499b-6r6r7\" (UID: \"0ed83167-16c8-417b-9f27-077eefaf338b\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"
Sep 11 18:21:17.910420 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:17.910395    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ed83167-16c8-417b-9f27-077eefaf338b-cert\") pod \"lws-controller-manager-5588b8499b-6r6r7\" (UID: \"0ed83167-16c8-417b-9f27-077eefaf338b\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"
Sep 11 18:21:17.910529 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:17.910435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ed83167-16c8-417b-9f27-077eefaf338b-manager-config\") pod \"lws-controller-manager-5588b8499b-6r6r7\" (UID: \"0ed83167-16c8-417b-9f27-077eefaf338b\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"
Sep 11 18:21:18.011663 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:18.011630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4d2v\" (UniqueName: \"kubernetes.io/projected/0ed83167-16c8-417b-9f27-077eefaf338b-kube-api-access-z4d2v\") pod \"lws-controller-manager-5588b8499b-6r6r7\" (UID: \"0ed83167-16c8-417b-9f27-077eefaf338b\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"
Sep 11 18:21:18.011817 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:18.011686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ed83167-16c8-417b-9f27-077eefaf338b-cert\") pod \"lws-controller-manager-5588b8499b-6r6r7\" (UID: \"0ed83167-16c8-417b-9f27-077eefaf338b\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"
Sep 11 18:21:18.011817 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:18.011800    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ed83167-16c8-417b-9f27-077eefaf338b-manager-config\") pod \"lws-controller-manager-5588b8499b-6r6r7\" (UID: \"0ed83167-16c8-417b-9f27-077eefaf338b\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"
Sep 11 18:21:18.011900 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:18.011860    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ed83167-16c8-417b-9f27-077eefaf338b-metrics-cert\") pod \"lws-controller-manager-5588b8499b-6r6r7\" (UID: \"0ed83167-16c8-417b-9f27-077eefaf338b\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"
Sep 11 18:21:18.012438 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:18.012387    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ed83167-16c8-417b-9f27-077eefaf338b-manager-config\") pod \"lws-controller-manager-5588b8499b-6r6r7\" (UID: \"0ed83167-16c8-417b-9f27-077eefaf338b\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"
Sep 11 18:21:18.014459 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:18.014436    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ed83167-16c8-417b-9f27-077eefaf338b-metrics-cert\") pod \"lws-controller-manager-5588b8499b-6r6r7\" (UID: \"0ed83167-16c8-417b-9f27-077eefaf338b\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"
Sep 11 18:21:18.014571 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:18.014446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ed83167-16c8-417b-9f27-077eefaf338b-cert\") pod \"lws-controller-manager-5588b8499b-6r6r7\" (UID: \"0ed83167-16c8-417b-9f27-077eefaf338b\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"
Sep 11 18:21:18.021292 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:18.021270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4d2v\" (UniqueName: \"kubernetes.io/projected/0ed83167-16c8-417b-9f27-077eefaf338b-kube-api-access-z4d2v\") pod \"lws-controller-manager-5588b8499b-6r6r7\" (UID: \"0ed83167-16c8-417b-9f27-077eefaf338b\") " pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"
Sep 11 18:21:18.141059 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:18.141027    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"
Sep 11 18:21:18.431954 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:18.431821    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"]
Sep 11 18:21:18.434269 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:21:18.434235    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed83167_16c8_417b_9f27_077eefaf338b.slice/crio-728ebf15c82bb90b3b9940e79f8fa4fea43b373b0fc83ea610d4a41b389b7655 WatchSource:0}: Error finding container 728ebf15c82bb90b3b9940e79f8fa4fea43b373b0fc83ea610d4a41b389b7655: Status 404 returned error can't find the container with id 728ebf15c82bb90b3b9940e79f8fa4fea43b373b0fc83ea610d4a41b389b7655
Sep 11 18:21:18.436119 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:18.436104    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:21:19.323672 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:19.323639    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7" event={"ID":"0ed83167-16c8-417b-9f27-077eefaf338b","Type":"ContainerStarted","Data":"728ebf15c82bb90b3b9940e79f8fa4fea43b373b0fc83ea610d4a41b389b7655"}
Sep 11 18:21:20.849177 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:20.849141    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"]
Sep 11 18:21:20.851495 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:20.851455    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:20.853766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:20.853719    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 18:21:20.853995 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:20.853977    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8v878\""
Sep 11 18:21:20.854066 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:20.854042    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 18:21:20.854205 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:20.854187    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 18:21:20.854404 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:20.854392    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 18:21:20.870193 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:20.870170    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"]
Sep 11 18:21:20.938164 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:20.938130    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7059e104-48c1-4026-b701-eceb3e64f340-tmp\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-lm8nq\" (UID: \"7059e104-48c1-4026-b701-eceb3e64f340\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:20.938317 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:20.938197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7059e104-48c1-4026-b701-eceb3e64f340-manifests\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-lm8nq\" (UID: \"7059e104-48c1-4026-b701-eceb3e64f340\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:20.938317 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:20.938217    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7059e104-48c1-4026-b701-eceb3e64f340-webhook-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-lm8nq\" (UID: \"7059e104-48c1-4026-b701-eceb3e64f340\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:20.938317 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:20.938243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7059e104-48c1-4026-b701-eceb3e64f340-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-lm8nq\" (UID: \"7059e104-48c1-4026-b701-eceb3e64f340\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:20.938317 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:20.938307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fd5\" (UniqueName: \"kubernetes.io/projected/7059e104-48c1-4026-b701-eceb3e64f340-kube-api-access-s7fd5\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-lm8nq\" (UID: \"7059e104-48c1-4026-b701-eceb3e64f340\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:21.039788 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:21.039703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7059e104-48c1-4026-b701-eceb3e64f340-tmp\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-lm8nq\" (UID: \"7059e104-48c1-4026-b701-eceb3e64f340\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:21.039788 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:21.039753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7059e104-48c1-4026-b701-eceb3e64f340-manifests\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-lm8nq\" (UID: \"7059e104-48c1-4026-b701-eceb3e64f340\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:21.039788 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:21.039774    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7059e104-48c1-4026-b701-eceb3e64f340-webhook-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-lm8nq\" (UID: \"7059e104-48c1-4026-b701-eceb3e64f340\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:21.040005 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:21.039799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7059e104-48c1-4026-b701-eceb3e64f340-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-lm8nq\" (UID: \"7059e104-48c1-4026-b701-eceb3e64f340\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:21.040005 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:21.039851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fd5\" (UniqueName: \"kubernetes.io/projected/7059e104-48c1-4026-b701-eceb3e64f340-kube-api-access-s7fd5\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-lm8nq\" (UID: \"7059e104-48c1-4026-b701-eceb3e64f340\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:21.040180 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:21.040148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7059e104-48c1-4026-b701-eceb3e64f340-manifests\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-lm8nq\" (UID: \"7059e104-48c1-4026-b701-eceb3e64f340\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:21.042391 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:21.042362    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7059e104-48c1-4026-b701-eceb3e64f340-tmp\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-lm8nq\" (UID: \"7059e104-48c1-4026-b701-eceb3e64f340\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:21.042509 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:21.042442    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7059e104-48c1-4026-b701-eceb3e64f340-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-lm8nq\" (UID: \"7059e104-48c1-4026-b701-eceb3e64f340\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:21.042573 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:21.042558    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7059e104-48c1-4026-b701-eceb3e64f340-webhook-cert\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-lm8nq\" (UID: \"7059e104-48c1-4026-b701-eceb3e64f340\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:21.055795 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:21.055776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fd5\" (UniqueName: \"kubernetes.io/projected/7059e104-48c1-4026-b701-eceb3e64f340-kube-api-access-s7fd5\") pod \"opendatahub-operator-controller-manager-76cff9b6b6-lm8nq\" (UID: \"7059e104-48c1-4026-b701-eceb3e64f340\") " pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:21.162119 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:21.162086    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:21.336196 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:21.336104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7" event={"ID":"0ed83167-16c8-417b-9f27-077eefaf338b","Type":"ContainerStarted","Data":"5bb48738213f4e362ae135d162956cb4d1908649501a06f9134200a92604018a"}
Sep 11 18:21:21.336196 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:21.336175    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"
Sep 11 18:21:21.353881 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:21.353825    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7" podStartSLOduration=2.027801107 podStartE2EDuration="4.353804734s" totalImagesPullingTime="2.326003627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:21.353669731 +0000 UTC m=+358.094796576" watchObservedRunningTime="2026-09-11 18:21:21.353804734 +0000 UTC m=+358.094931579"
Sep 11 18:21:21.436826 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:21.436804    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"]
Sep 11 18:21:21.439120 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:21:21.439093    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7059e104_48c1_4026_b701_eceb3e64f340.slice/crio-48bfb72223f16e702ddf81798d2d5814b4ae2b5ad67f2214c51d9193c8230f13 WatchSource:0}: Error finding container 48bfb72223f16e702ddf81798d2d5814b4ae2b5ad67f2214c51d9193c8230f13: Status 404 returned error can't find the container with id 48bfb72223f16e702ddf81798d2d5814b4ae2b5ad67f2214c51d9193c8230f13
Sep 11 18:21:22.345186 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:22.345142    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq" event={"ID":"7059e104-48c1-4026-b701-eceb3e64f340","Type":"ContainerStarted","Data":"48bfb72223f16e702ddf81798d2d5814b4ae2b5ad67f2214c51d9193c8230f13"}
Sep 11 18:21:25.361415 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:25.361378    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq" event={"ID":"7059e104-48c1-4026-b701-eceb3e64f340","Type":"ContainerStarted","Data":"8d8460dc2faab74abb33567c65b99075200b9a91edf09d148e5bd83ec6f36fb5"}
Sep 11 18:21:26.367564 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:26.367525    2581 generic.go:356] "Generic (PLEG): container finished" podID="7059e104-48c1-4026-b701-eceb3e64f340" containerID="8d8460dc2faab74abb33567c65b99075200b9a91edf09d148e5bd83ec6f36fb5" exitCode=0
Sep 11 18:21:26.367924 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:26.367604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq" event={"ID":"7059e104-48c1-4026-b701-eceb3e64f340","Type":"ContainerDied","Data":"8d8460dc2faab74abb33567c65b99075200b9a91edf09d148e5bd83ec6f36fb5"}
Sep 11 18:21:27.373721 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:27.373635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq" event={"ID":"7059e104-48c1-4026-b701-eceb3e64f340","Type":"ContainerStarted","Data":"a04e862e7a31536992959b18a33e55ca1190bd4f4bcdc56ab8e1adab4cdbde29"}
Sep 11 18:21:27.373721 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:27.373710    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:27.404246 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:27.404115    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq" podStartSLOduration=2.513932106 podStartE2EDuration="7.404097195s" totalImagesPullingTime="3.809791847s" totalInitContainerRuntime="1.080373242s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:27.403522474 +0000 UTC m=+364.144649313" watchObservedRunningTime="2026-09-11 18:21:27.404097195 +0000 UTC m=+364.145224038"
Sep 11 18:21:32.347468 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:32.347433    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5588b8499b-6r6r7"
Sep 11 18:21:38.380056 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:38.380023    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76cff9b6b6-lm8nq"
Sep 11 18:21:50.877616 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:50.877579    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj"]
Sep 11 18:21:50.881072 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:50.881047    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj"
Sep 11 18:21:50.883261 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:50.883240    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 18:21:50.883987 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:50.883973    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 18:21:50.884063 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:50.884045    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-slcmn\""
Sep 11 18:21:50.893134 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:50.893110    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj"]
Sep 11 18:21:50.983399 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:50.983366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrcwn\" (UniqueName: \"kubernetes.io/projected/917d39f0-df45-4535-8f45-273b74114c36-kube-api-access-zrcwn\") pod \"kube-auth-proxy-67fd44999f-9nfjj\" (UID: \"917d39f0-df45-4535-8f45-273b74114c36\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj"
Sep 11 18:21:50.983399 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:50.983404    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/917d39f0-df45-4535-8f45-273b74114c36-tls-certs\") pod \"kube-auth-proxy-67fd44999f-9nfjj\" (UID: \"917d39f0-df45-4535-8f45-273b74114c36\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj"
Sep 11 18:21:50.983593 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:50.983424    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/917d39f0-df45-4535-8f45-273b74114c36-tmp\") pod \"kube-auth-proxy-67fd44999f-9nfjj\" (UID: \"917d39f0-df45-4535-8f45-273b74114c36\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj"
Sep 11 18:21:51.084191 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:51.084159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrcwn\" (UniqueName: \"kubernetes.io/projected/917d39f0-df45-4535-8f45-273b74114c36-kube-api-access-zrcwn\") pod \"kube-auth-proxy-67fd44999f-9nfjj\" (UID: \"917d39f0-df45-4535-8f45-273b74114c36\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj"
Sep 11 18:21:51.084191 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:51.084196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/917d39f0-df45-4535-8f45-273b74114c36-tls-certs\") pod \"kube-auth-proxy-67fd44999f-9nfjj\" (UID: \"917d39f0-df45-4535-8f45-273b74114c36\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj"
Sep 11 18:21:51.084383 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:51.084216    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/917d39f0-df45-4535-8f45-273b74114c36-tmp\") pod \"kube-auth-proxy-67fd44999f-9nfjj\" (UID: \"917d39f0-df45-4535-8f45-273b74114c36\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj"
Sep 11 18:21:51.086525 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:51.086505    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/917d39f0-df45-4535-8f45-273b74114c36-tmp\") pod \"kube-auth-proxy-67fd44999f-9nfjj\" (UID: \"917d39f0-df45-4535-8f45-273b74114c36\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj"
Sep 11 18:21:51.086753 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:51.086736    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/917d39f0-df45-4535-8f45-273b74114c36-tls-certs\") pod \"kube-auth-proxy-67fd44999f-9nfjj\" (UID: \"917d39f0-df45-4535-8f45-273b74114c36\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj"
Sep 11 18:21:51.095674 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:51.095644    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrcwn\" (UniqueName: \"kubernetes.io/projected/917d39f0-df45-4535-8f45-273b74114c36-kube-api-access-zrcwn\") pod \"kube-auth-proxy-67fd44999f-9nfjj\" (UID: \"917d39f0-df45-4535-8f45-273b74114c36\") " pod="openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj"
Sep 11 18:21:51.193440 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:51.193351    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj"
Sep 11 18:21:51.469585 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:51.469559    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj"]
Sep 11 18:21:51.472316 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:21:51.472269    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917d39f0_df45_4535_8f45_273b74114c36.slice/crio-88bf016e1e74d6f86f38cce5d6f86f784c11f10b6671d05d2ea60fd566e5cdde WatchSource:0}: Error finding container 88bf016e1e74d6f86f38cce5d6f86f784c11f10b6671d05d2ea60fd566e5cdde: Status 404 returned error can't find the container with id 88bf016e1e74d6f86f38cce5d6f86f784c11f10b6671d05d2ea60fd566e5cdde
Sep 11 18:21:52.483377 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:52.483333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj" event={"ID":"917d39f0-df45-4535-8f45-273b74114c36","Type":"ContainerStarted","Data":"88bf016e1e74d6f86f38cce5d6f86f784c11f10b6671d05d2ea60fd566e5cdde"}
Sep 11 18:21:54.493868 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:54.493833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj" event={"ID":"917d39f0-df45-4535-8f45-273b74114c36","Type":"ContainerStarted","Data":"b2fbc7dc28faad9a46e2f7a6b90085273f34d638479e455ff7cabc4cd658cdf8"}
Sep 11 18:21:54.512186 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:21:54.512132    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-67fd44999f-9nfjj" podStartSLOduration=1.873986281 podStartE2EDuration="4.512111674s" totalImagesPullingTime="2.638125393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:54.511428228 +0000 UTC m=+391.252555072" watchObservedRunningTime="2026-09-11 18:21:54.512111674 +0000 UTC m=+391.253238521"
Sep 11 18:23:16.745303 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:16.745271    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-56gf4"]
Sep 11 18:23:16.748511 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:16.748471    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-56gf4"
Sep 11 18:23:16.750876 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:16.750835    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 18:23:16.751015 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:16.750934    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 18:23:16.751015 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:16.750958    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 18:23:16.751015 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:16.750964    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-zkpt7\""
Sep 11 18:23:16.757712 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:16.757691    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-56gf4"]
Sep 11 18:23:16.901317 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:16.901287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s594\" (UniqueName: \"kubernetes.io/projected/b2ba9e8b-1e6f-4a80-8b0c-39f09e1fb8be-kube-api-access-8s594\") pod \"dns-operator-controller-manager-7b84cd7c9c-56gf4\" (UID: \"b2ba9e8b-1e6f-4a80-8b0c-39f09e1fb8be\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-56gf4"
Sep 11 18:23:17.001991 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:17.001930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s594\" (UniqueName: \"kubernetes.io/projected/b2ba9e8b-1e6f-4a80-8b0c-39f09e1fb8be-kube-api-access-8s594\") pod \"dns-operator-controller-manager-7b84cd7c9c-56gf4\" (UID: \"b2ba9e8b-1e6f-4a80-8b0c-39f09e1fb8be\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-56gf4"
Sep 11 18:23:17.010617 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:17.010599    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s594\" (UniqueName: \"kubernetes.io/projected/b2ba9e8b-1e6f-4a80-8b0c-39f09e1fb8be-kube-api-access-8s594\") pod \"dns-operator-controller-manager-7b84cd7c9c-56gf4\" (UID: \"b2ba9e8b-1e6f-4a80-8b0c-39f09e1fb8be\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-56gf4"
Sep 11 18:23:17.058448 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:17.058430    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-56gf4"
Sep 11 18:23:17.337025 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:17.336888    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-56gf4"]
Sep 11 18:23:17.340381 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:23:17.340351    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ba9e8b_1e6f_4a80_8b0c_39f09e1fb8be.slice/crio-cf2e7d1e747d76cddf0035994e7882d961891c5b1fa41cf9a346a854abc17901 WatchSource:0}: Error finding container cf2e7d1e747d76cddf0035994e7882d961891c5b1fa41cf9a346a854abc17901: Status 404 returned error can't find the container with id cf2e7d1e747d76cddf0035994e7882d961891c5b1fa41cf9a346a854abc17901
Sep 11 18:23:17.851698 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:17.851661    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-56gf4" event={"ID":"b2ba9e8b-1e6f-4a80-8b0c-39f09e1fb8be","Type":"ContainerStarted","Data":"cf2e7d1e747d76cddf0035994e7882d961891c5b1fa41cf9a346a854abc17901"}
Sep 11 18:23:20.869745 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:20.869710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-56gf4" event={"ID":"b2ba9e8b-1e6f-4a80-8b0c-39f09e1fb8be","Type":"ContainerStarted","Data":"5731c4f9dad8a7752307f2664063a0308730e05dbc144f3fbe8331b9f53275de"}
Sep 11 18:23:20.870170 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:20.869838    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-56gf4"
Sep 11 18:23:20.893671 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:20.893625    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-56gf4" podStartSLOduration=2.269159881 podStartE2EDuration="4.893610028s" totalImagesPullingTime="2.624450147s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:20.892098386 +0000 UTC m=+477.633225230" watchObservedRunningTime="2026-09-11 18:23:20.893610028 +0000 UTC m=+477.634736871"
Sep 11 18:23:21.950356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:21.950320    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7"]
Sep 11 18:23:21.953612 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:21.953596    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7"
Sep 11 18:23:21.955773 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:21.955749    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zr6cv\""
Sep 11 18:23:21.955904 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:21.955822    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 11 18:23:21.956069 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:21.956052    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 11 18:23:21.967302 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:21.963660    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7"]
Sep 11 18:23:22.044078 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:22.044046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/edaa18e3-d95c-40f2-84d7-7271b9921e11-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-n6nh7\" (UID: \"edaa18e3-d95c-40f2-84d7-7271b9921e11\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7"
Sep 11 18:23:22.044248 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:22.044139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5lwf\" (UniqueName: \"kubernetes.io/projected/edaa18e3-d95c-40f2-84d7-7271b9921e11-kube-api-access-k5lwf\") pod \"kuadrant-console-plugin-fbdd4f445-n6nh7\" (UID: \"edaa18e3-d95c-40f2-84d7-7271b9921e11\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7"
Sep 11 18:23:22.044248 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:22.044175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/edaa18e3-d95c-40f2-84d7-7271b9921e11-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-n6nh7\" (UID: \"edaa18e3-d95c-40f2-84d7-7271b9921e11\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7"
Sep 11 18:23:22.145458 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:22.145423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/edaa18e3-d95c-40f2-84d7-7271b9921e11-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-n6nh7\" (UID: \"edaa18e3-d95c-40f2-84d7-7271b9921e11\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7"
Sep 11 18:23:22.145646 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:22.145511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5lwf\" (UniqueName: \"kubernetes.io/projected/edaa18e3-d95c-40f2-84d7-7271b9921e11-kube-api-access-k5lwf\") pod \"kuadrant-console-plugin-fbdd4f445-n6nh7\" (UID: \"edaa18e3-d95c-40f2-84d7-7271b9921e11\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7"
Sep 11 18:23:22.145646 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:22.145548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/edaa18e3-d95c-40f2-84d7-7271b9921e11-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-n6nh7\" (UID: \"edaa18e3-d95c-40f2-84d7-7271b9921e11\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7"
Sep 11 18:23:22.146175 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:22.146150    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/edaa18e3-d95c-40f2-84d7-7271b9921e11-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-n6nh7\" (UID: \"edaa18e3-d95c-40f2-84d7-7271b9921e11\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7"
Sep 11 18:23:22.147881 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:22.147851    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/edaa18e3-d95c-40f2-84d7-7271b9921e11-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-n6nh7\" (UID: \"edaa18e3-d95c-40f2-84d7-7271b9921e11\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7"
Sep 11 18:23:22.153882 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:22.153863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5lwf\" (UniqueName: \"kubernetes.io/projected/edaa18e3-d95c-40f2-84d7-7271b9921e11-kube-api-access-k5lwf\") pod \"kuadrant-console-plugin-fbdd4f445-n6nh7\" (UID: \"edaa18e3-d95c-40f2-84d7-7271b9921e11\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7"
Sep 11 18:23:22.281350 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:22.281285    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7"
Sep 11 18:23:22.552819 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:22.552796    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7"]
Sep 11 18:23:22.555018 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:23:22.554990    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedaa18e3_d95c_40f2_84d7_7271b9921e11.slice/crio-0952b79f9c314e95563ed5e2644f28c43541c63ca220d6071c5d7c49ef5a5612 WatchSource:0}: Error finding container 0952b79f9c314e95563ed5e2644f28c43541c63ca220d6071c5d7c49ef5a5612: Status 404 returned error can't find the container with id 0952b79f9c314e95563ed5e2644f28c43541c63ca220d6071c5d7c49ef5a5612
Sep 11 18:23:22.878942 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:22.878911    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7" event={"ID":"edaa18e3-d95c-40f2-84d7-7271b9921e11","Type":"ContainerStarted","Data":"0952b79f9c314e95563ed5e2644f28c43541c63ca220d6071c5d7c49ef5a5612"}
Sep 11 18:23:27.905522 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:27.905464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7" event={"ID":"edaa18e3-d95c-40f2-84d7-7271b9921e11","Type":"ContainerStarted","Data":"9545636af4882fe265baf93360715e0a286d430b1942e93b8e10dbebd6b6fec1"}
Sep 11 18:23:27.922209 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:27.922161    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n6nh7" podStartSLOduration=2.494829052 podStartE2EDuration="6.922145377s" totalImagesPullingTime="4.427316325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:27.920623929 +0000 UTC m=+484.661750773" watchObservedRunningTime="2026-09-11 18:23:27.922145377 +0000 UTC m=+484.663272220"
Sep 11 18:23:31.875840 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:23:31.875809    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-56gf4"
Sep 11 18:24:02.937145 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:02.937108    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:24:02.940179 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:02.940162    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"
Sep 11 18:24:02.942197 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:02.942175    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 11 18:24:02.947876 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:02.947852    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:24:02.983357 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:02.983334    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:24:03.061572 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:03.061538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/edff1179-2a27-4a36-9373-46cab7d17a9f-config-file\") pod \"limitador-limitador-84994f7ffc-mkgcd\" (UID: \"edff1179-2a27-4a36-9373-46cab7d17a9f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"
Sep 11 18:24:03.061725 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:03.061602    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkgcr\" (UniqueName: \"kubernetes.io/projected/edff1179-2a27-4a36-9373-46cab7d17a9f-kube-api-access-hkgcr\") pod \"limitador-limitador-84994f7ffc-mkgcd\" (UID: \"edff1179-2a27-4a36-9373-46cab7d17a9f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"
Sep 11 18:24:03.162758 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:03.162724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/edff1179-2a27-4a36-9373-46cab7d17a9f-config-file\") pod \"limitador-limitador-84994f7ffc-mkgcd\" (UID: \"edff1179-2a27-4a36-9373-46cab7d17a9f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"
Sep 11 18:24:03.162870 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:03.162790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkgcr\" (UniqueName: \"kubernetes.io/projected/edff1179-2a27-4a36-9373-46cab7d17a9f-kube-api-access-hkgcr\") pod \"limitador-limitador-84994f7ffc-mkgcd\" (UID: \"edff1179-2a27-4a36-9373-46cab7d17a9f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"
Sep 11 18:24:03.163350 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:03.163331    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/edff1179-2a27-4a36-9373-46cab7d17a9f-config-file\") pod \"limitador-limitador-84994f7ffc-mkgcd\" (UID: \"edff1179-2a27-4a36-9373-46cab7d17a9f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"
Sep 11 18:24:03.172931 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:03.172901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkgcr\" (UniqueName: \"kubernetes.io/projected/edff1179-2a27-4a36-9373-46cab7d17a9f-kube-api-access-hkgcr\") pod \"limitador-limitador-84994f7ffc-mkgcd\" (UID: \"edff1179-2a27-4a36-9373-46cab7d17a9f\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"
Sep 11 18:24:03.250647 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:03.250581    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"
Sep 11 18:24:03.525524 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:03.525497    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:24:03.527648 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:24:03.527622    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedff1179_2a27_4a36_9373_46cab7d17a9f.slice/crio-7ac90d964b936d5a4e1c12c0cbfcfcf23bf12cc51624d577e45fbf8d2367f14d WatchSource:0}: Error finding container 7ac90d964b936d5a4e1c12c0cbfcfcf23bf12cc51624d577e45fbf8d2367f14d: Status 404 returned error can't find the container with id 7ac90d964b936d5a4e1c12c0cbfcfcf23bf12cc51624d577e45fbf8d2367f14d
Sep 11 18:24:04.064549 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:04.064512    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-mkgcd" event={"ID":"edff1179-2a27-4a36-9373-46cab7d17a9f","Type":"ContainerStarted","Data":"7ac90d964b936d5a4e1c12c0cbfcfcf23bf12cc51624d577e45fbf8d2367f14d"}
Sep 11 18:24:05.070761 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:05.070673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-mkgcd" event={"ID":"edff1179-2a27-4a36-9373-46cab7d17a9f","Type":"ContainerStarted","Data":"382bf533b3b99ea3548f88757d35b70ca855a8a1f916a8f56712f6c92616da22"}
Sep 11 18:24:05.071110 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:05.070772    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"
Sep 11 18:24:05.088116 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:05.088072    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-mkgcd" podStartSLOduration=1.8228351200000001 podStartE2EDuration="3.088058789s" totalImagesPullingTime="1.265223669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:05.086463646 +0000 UTC m=+521.827590488" watchObservedRunningTime="2026-09-11 18:24:05.088058789 +0000 UTC m=+521.829185632"
Sep 11 18:24:16.076309 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:24:16.076229    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"
Sep 11 18:25:15.832128 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:15.832092    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-2292w"]
Sep 11 18:25:15.835306 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:15.835291    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-2292w"
Sep 11 18:25:15.837498 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:15.837452    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-j5wvs\""
Sep 11 18:25:15.837619 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:15.837548    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 18:25:15.846006 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:15.845983    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-2292w"]
Sep 11 18:25:15.894784 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:15.894755    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:25:15.971671 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:15.971636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc357e4e-df13-4296-84eb-8d0d846b34be-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-2292w\" (UID: \"cc357e4e-df13-4296-84eb-8d0d846b34be\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2292w"
Sep 11 18:25:15.971826 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:15.971679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7vcm\" (UniqueName: \"kubernetes.io/projected/cc357e4e-df13-4296-84eb-8d0d846b34be-kube-api-access-k7vcm\") pod \"payload-processing-5b54b7fbb7-2292w\" (UID: \"cc357e4e-df13-4296-84eb-8d0d846b34be\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2292w"
Sep 11 18:25:16.072887 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:16.072848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc357e4e-df13-4296-84eb-8d0d846b34be-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-2292w\" (UID: \"cc357e4e-df13-4296-84eb-8d0d846b34be\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2292w"
Sep 11 18:25:16.073059 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:16.072902    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7vcm\" (UniqueName: \"kubernetes.io/projected/cc357e4e-df13-4296-84eb-8d0d846b34be-kube-api-access-k7vcm\") pod \"payload-processing-5b54b7fbb7-2292w\" (UID: \"cc357e4e-df13-4296-84eb-8d0d846b34be\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2292w"
Sep 11 18:25:16.073619 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:16.073602    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc357e4e-df13-4296-84eb-8d0d846b34be-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-2292w\" (UID: \"cc357e4e-df13-4296-84eb-8d0d846b34be\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2292w"
Sep 11 18:25:16.081340 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:16.081316    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7vcm\" (UniqueName: \"kubernetes.io/projected/cc357e4e-df13-4296-84eb-8d0d846b34be-kube-api-access-k7vcm\") pod \"payload-processing-5b54b7fbb7-2292w\" (UID: \"cc357e4e-df13-4296-84eb-8d0d846b34be\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-2292w"
Sep 11 18:25:16.145707 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:16.145689    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-2292w"
Sep 11 18:25:16.415460 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:16.415436    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-2292w"]
Sep 11 18:25:16.417086 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:25:16.417058    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc357e4e_df13_4296_84eb_8d0d846b34be.slice/crio-748eda9168f943c999801b9404f29c6bc2b3878b32789999fd2a96766911cd7d WatchSource:0}: Error finding container 748eda9168f943c999801b9404f29c6bc2b3878b32789999fd2a96766911cd7d: Status 404 returned error can't find the container with id 748eda9168f943c999801b9404f29c6bc2b3878b32789999fd2a96766911cd7d
Sep 11 18:25:17.396241 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:17.396203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-2292w" event={"ID":"cc357e4e-df13-4296-84eb-8d0d846b34be","Type":"ContainerStarted","Data":"748eda9168f943c999801b9404f29c6bc2b3878b32789999fd2a96766911cd7d"}
Sep 11 18:25:19.406530 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:19.406497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-2292w" event={"ID":"cc357e4e-df13-4296-84eb-8d0d846b34be","Type":"ContainerStarted","Data":"ab18634b3fe04b6072c08f6b03986e2eb147434b88d1016612701a63dadedc93"}
Sep 11 18:25:19.406981 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:19.406560    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-2292w"
Sep 11 18:25:19.423011 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:19.422955    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-2292w" podStartSLOduration=2.066502604 podStartE2EDuration="4.422935341s" totalImagesPullingTime="2.356432737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:19.421246607 +0000 UTC m=+596.162373455" watchObservedRunningTime="2026-09-11 18:25:19.422935341 +0000 UTC m=+596.164062185"
Sep 11 18:25:30.413660 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:30.413627    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-2292w"
Sep 11 18:25:30.564559 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:30.564519    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:25:41.966642 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:41.966610    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:25:47.248800 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:47.248727    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:25:57.342572 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.342536    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"]
Sep 11 18:25:57.347213 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.347195    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.349340 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.349322    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 18:25:57.349340 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.349330    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 18:25:57.349952 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.349932    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zv74c\""
Sep 11 18:25:57.350046 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.350034    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 11 18:25:57.354002 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.353982    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"]
Sep 11 18:25:57.429760 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.429724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pg6c\" (UniqueName: \"kubernetes.io/projected/3d9faf35-7782-4f47-8924-13f951123424-kube-api-access-8pg6c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.429760 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.429763    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9faf35-7782-4f47-8924-13f951123424-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.429939 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.429788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9faf35-7782-4f47-8924-13f951123424-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.429939 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.429836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9faf35-7782-4f47-8924-13f951123424-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.429939 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.429881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9faf35-7782-4f47-8924-13f951123424-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.429939 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.429904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9faf35-7782-4f47-8924-13f951123424-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.530454 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.530421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pg6c\" (UniqueName: \"kubernetes.io/projected/3d9faf35-7782-4f47-8924-13f951123424-kube-api-access-8pg6c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.530608 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.530501    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9faf35-7782-4f47-8924-13f951123424-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.530608 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.530538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9faf35-7782-4f47-8924-13f951123424-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.530608 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.530560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9faf35-7782-4f47-8924-13f951123424-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.530608 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.530585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9faf35-7782-4f47-8924-13f951123424-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.530608 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.530606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9faf35-7782-4f47-8924-13f951123424-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.530897 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.530878    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d9faf35-7782-4f47-8924-13f951123424-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.530988 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.530966    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d9faf35-7782-4f47-8924-13f951123424-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.531038 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.531015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d9faf35-7782-4f47-8924-13f951123424-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.532891 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.532869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d9faf35-7782-4f47-8924-13f951123424-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.533088 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.533068    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d9faf35-7782-4f47-8924-13f951123424-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.538328 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.538307    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pg6c\" (UniqueName: \"kubernetes.io/projected/3d9faf35-7782-4f47-8924-13f951123424-kube-api-access-8pg6c\") pod \"e2e-trlp-test-simulated-kserve-598466548d-f7mc2\" (UID: \"3d9faf35-7782-4f47-8924-13f951123424\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.658068 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.658036    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:25:57.930213 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:57.930054    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"]
Sep 11 18:25:57.932430 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:25:57.932402    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d9faf35_7782_4f47_8924_13f951123424.slice/crio-60f2e68459db36d70b27bec37baa75b491820e439476d204dfb25e4fd42dcb55 WatchSource:0}: Error finding container 60f2e68459db36d70b27bec37baa75b491820e439476d204dfb25e4fd42dcb55: Status 404 returned error can't find the container with id 60f2e68459db36d70b27bec37baa75b491820e439476d204dfb25e4fd42dcb55
Sep 11 18:25:58.546763 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:58.546729    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:25:58.583524 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:58.583491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2" event={"ID":"3d9faf35-7782-4f47-8924-13f951123424","Type":"ContainerStarted","Data":"60f2e68459db36d70b27bec37baa75b491820e439476d204dfb25e4fd42dcb55"}
Sep 11 18:25:59.590015 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:59.589977    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2" event={"ID":"3d9faf35-7782-4f47-8924-13f951123424","Type":"ContainerStarted","Data":"92a651bd227ac143ddf3d8661e684c25ba20b2ec213e474ef31cb39759f92098"}
Sep 11 18:25:59.608294 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:25:59.608245    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2" podStartSLOduration=1.603983041 podStartE2EDuration="2.608228376s" totalImagesPullingTime="1.004245335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:59.606601801 +0000 UTC m=+636.347728645" watchObservedRunningTime="2026-09-11 18:25:59.608228376 +0000 UTC m=+636.349355215"
Sep 11 18:26:06.746881 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:06.746847    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:26:07.658991 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:07.658957    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:26:07.659161 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:07.659022    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:26:07.671313 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:07.671284    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:26:08.648450 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:08.648415    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-f7mc2"
Sep 11 18:26:13.847294 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:13.847260    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:26:21.227675 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.227638    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"]
Sep 11 18:26:21.231939 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.231919    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.233889 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.233874    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 11 18:26:21.239330 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.239301    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"]
Sep 11 18:26:21.336998 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.336963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.336998 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.336993    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.337207 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.337032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.337207 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.337047    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.337207 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.337085    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gnck\" (UniqueName: \"kubernetes.io/projected/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-kube-api-access-6gnck\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.337207 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.337125    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.437773 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.437742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gnck\" (UniqueName: \"kubernetes.io/projected/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-kube-api-access-6gnck\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.437905 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.437801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.437982 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.437959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.438023 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.437993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.438023 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.438020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.438146 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.438036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.438330 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.438308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.438393 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.438351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.438448 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.438428    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.440300 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.440271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.440507 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.440491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.445990 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.445967    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gnck\" (UniqueName: \"kubernetes.io/projected/754e8a1e-d3bf-4d20-b0d6-3ae61a880424-kube-api-access-6gnck\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w8w48\" (UID: \"754e8a1e-d3bf-4d20-b0d6-3ae61a880424\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.543274 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.543187    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:21.817368 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.817341    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"]
Sep 11 18:26:21.820047 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:26:21.820017    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod754e8a1e_d3bf_4d20_b0d6_3ae61a880424.slice/crio-78f89f210415a1e6061093aae872f06e0a7e61e8d680501c33402df3939aae7c WatchSource:0}: Error finding container 78f89f210415a1e6061093aae872f06e0a7e61e8d680501c33402df3939aae7c: Status 404 returned error can't find the container with id 78f89f210415a1e6061093aae872f06e0a7e61e8d680501c33402df3939aae7c
Sep 11 18:26:21.821334 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:21.821315    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:26:22.704767 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:22.704729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48" event={"ID":"754e8a1e-d3bf-4d20-b0d6-3ae61a880424","Type":"ContainerStarted","Data":"5922fb3a5726415a9027258a8538819a2c81ced6d16237a8f460f8cd407234b5"}
Sep 11 18:26:22.704767 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:22.704772    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48" event={"ID":"754e8a1e-d3bf-4d20-b0d6-3ae61a880424","Type":"ContainerStarted","Data":"78f89f210415a1e6061093aae872f06e0a7e61e8d680501c33402df3939aae7c"}
Sep 11 18:26:22.722308 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:22.722260    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48" podStartSLOduration=1.537172568 podStartE2EDuration="1.722244222s" totalImagesPullingTime="185.071654ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:22.720386272 +0000 UTC m=+659.461513121" watchObservedRunningTime="2026-09-11 18:26:22.722244222 +0000 UTC m=+659.463371064"
Sep 11 18:26:23.542524 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:23.542491    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:26:31.544372 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:31.544332    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:31.544372 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:31.544374    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:31.557022 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:31.556997    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:26:31.763330 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:26:31.763299    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w8w48"
Sep 11 18:27:33.721106 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:33.721068    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7647-dcdd43"]
Sep 11 18:27:33.724413 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:33.724393    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-dcdd43"
Sep 11 18:27:33.726416 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:33.726397    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bbqnn\""
Sep 11 18:27:33.729045 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:33.729022    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7647-dcdd43"]
Sep 11 18:27:33.863050 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:33.863013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxx4t\" (UniqueName: \"kubernetes.io/projected/e7a5c535-124b-44cc-bd62-820d6ee627e1-kube-api-access-mxx4t\") pod \"test-curl-7647-dcdd43\" (UID: \"e7a5c535-124b-44cc-bd62-820d6ee627e1\") " pod="opendatahub/test-curl-7647-dcdd43"
Sep 11 18:27:33.963811 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:33.963767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxx4t\" (UniqueName: \"kubernetes.io/projected/e7a5c535-124b-44cc-bd62-820d6ee627e1-kube-api-access-mxx4t\") pod \"test-curl-7647-dcdd43\" (UID: \"e7a5c535-124b-44cc-bd62-820d6ee627e1\") " pod="opendatahub/test-curl-7647-dcdd43"
Sep 11 18:27:33.972600 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:33.972540    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxx4t\" (UniqueName: \"kubernetes.io/projected/e7a5c535-124b-44cc-bd62-820d6ee627e1-kube-api-access-mxx4t\") pod \"test-curl-7647-dcdd43\" (UID: \"e7a5c535-124b-44cc-bd62-820d6ee627e1\") " pod="opendatahub/test-curl-7647-dcdd43"
Sep 11 18:27:34.034470 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:34.034442    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-dcdd43"
Sep 11 18:27:34.302077 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:34.302035    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7647-dcdd43"]
Sep 11 18:27:34.304552 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:27:34.304524    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7a5c535_124b_44cc_bd62_820d6ee627e1.slice/crio-c8435d1e488b93a04793ad35ddd4c12120fdf56b5a75c12e98e385affdb2ba20 WatchSource:0}: Error finding container c8435d1e488b93a04793ad35ddd4c12120fdf56b5a75c12e98e385affdb2ba20: Status 404 returned error can't find the container with id c8435d1e488b93a04793ad35ddd4c12120fdf56b5a75c12e98e385affdb2ba20
Sep 11 18:27:35.038748 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:35.038691    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7647-dcdd43" event={"ID":"e7a5c535-124b-44cc-bd62-820d6ee627e1","Type":"ContainerStarted","Data":"c8435d1e488b93a04793ad35ddd4c12120fdf56b5a75c12e98e385affdb2ba20"}
Sep 11 18:27:36.042273 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:36.042238    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:27:36.888703 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:36.888675    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"]
Sep 11 18:27:36.892114 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:36.892098    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"
Sep 11 18:27:36.893951 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:36.893931    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-efa8b9\""
Sep 11 18:27:36.902246 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:36.902227    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"]
Sep 11 18:27:36.915873 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:36.915852    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"]
Sep 11 18:27:36.919255 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:36.919241    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"
Sep 11 18:27:36.921208 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:36.921193    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-efa8b9-dockercfg-8zjlk\""
Sep 11 18:27:36.927400 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:36.927377    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"]
Sep 11 18:27:36.989457 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:36.989431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b432db-84df-4c1c-b999-a20494bb1663-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl\" (UID: \"12b432db-84df-4c1c-b999-a20494bb1663\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"
Sep 11 18:27:36.989572 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:36.989491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf2x2\" (UniqueName: \"kubernetes.io/projected/bebaebd3-07d3-45d4-b85e-364a006910b2-kube-api-access-kf2x2\") pod \"payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp\" (UID: \"bebaebd3-07d3-45d4-b85e-364a006910b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"
Sep 11 18:27:36.989572 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:36.989551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6q2b\" (UniqueName: \"kubernetes.io/projected/12b432db-84df-4c1c-b999-a20494bb1663-kube-api-access-w6q2b\") pod \"payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl\" (UID: \"12b432db-84df-4c1c-b999-a20494bb1663\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"
Sep 11 18:27:36.989647 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:36.989586    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebaebd3-07d3-45d4-b85e-364a006910b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp\" (UID: \"bebaebd3-07d3-45d4-b85e-364a006910b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"
Sep 11 18:27:37.050867 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.050793    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7647-dcdd43" event={"ID":"e7a5c535-124b-44cc-bd62-820d6ee627e1","Type":"ContainerStarted","Data":"7294aadad3cc47643bde6ecbe4f8fb6e4d2b1d638edbc1a3ab2126a3cf122a01"}
Sep 11 18:27:37.064686 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.064637    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7647-dcdd43" podStartSLOduration=1.568903657 podStartE2EDuration="4.064619086s" totalImagesPullingTime="2.495715429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:37.064136433 +0000 UTC m=+733.805263275" watchObservedRunningTime="2026-09-11 18:27:37.064619086 +0000 UTC m=+733.805745929"
Sep 11 18:27:37.090226 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.090202    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebaebd3-07d3-45d4-b85e-364a006910b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp\" (UID: \"bebaebd3-07d3-45d4-b85e-364a006910b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"
Sep 11 18:27:37.090314 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.090264    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b432db-84df-4c1c-b999-a20494bb1663-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl\" (UID: \"12b432db-84df-4c1c-b999-a20494bb1663\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"
Sep 11 18:27:37.090314 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.090301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf2x2\" (UniqueName: \"kubernetes.io/projected/bebaebd3-07d3-45d4-b85e-364a006910b2-kube-api-access-kf2x2\") pod \"payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp\" (UID: \"bebaebd3-07d3-45d4-b85e-364a006910b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"
Sep 11 18:27:37.090392 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.090321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6q2b\" (UniqueName: \"kubernetes.io/projected/12b432db-84df-4c1c-b999-a20494bb1663-kube-api-access-w6q2b\") pod \"payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl\" (UID: \"12b432db-84df-4c1c-b999-a20494bb1663\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"
Sep 11 18:27:37.090928 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.090910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebaebd3-07d3-45d4-b85e-364a006910b2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp\" (UID: \"bebaebd3-07d3-45d4-b85e-364a006910b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"
Sep 11 18:27:37.091013 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.090996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b432db-84df-4c1c-b999-a20494bb1663-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl\" (UID: \"12b432db-84df-4c1c-b999-a20494bb1663\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"
Sep 11 18:27:37.099081 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.099056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf2x2\" (UniqueName: \"kubernetes.io/projected/bebaebd3-07d3-45d4-b85e-364a006910b2-kube-api-access-kf2x2\") pod \"payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp\" (UID: \"bebaebd3-07d3-45d4-b85e-364a006910b2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"
Sep 11 18:27:37.099317 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.099297    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6q2b\" (UniqueName: \"kubernetes.io/projected/12b432db-84df-4c1c-b999-a20494bb1663-kube-api-access-w6q2b\") pod \"payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl\" (UID: \"12b432db-84df-4c1c-b999-a20494bb1663\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"
Sep 11 18:27:37.202159 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.202132    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"
Sep 11 18:27:37.229996 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.229975    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"
Sep 11 18:27:37.475174 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.475149    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"]
Sep 11 18:27:37.476915 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:27:37.476891    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b432db_84df_4c1c_b999_a20494bb1663.slice/crio-6a7d79b0c0e7a908c05179a6121839a28ec400c59cc4a688a149ff2b28fec14b WatchSource:0}: Error finding container 6a7d79b0c0e7a908c05179a6121839a28ec400c59cc4a688a149ff2b28fec14b: Status 404 returned error can't find the container with id 6a7d79b0c0e7a908c05179a6121839a28ec400c59cc4a688a149ff2b28fec14b
Sep 11 18:27:37.519607 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.519584    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"]
Sep 11 18:27:37.530595 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:27:37.530568    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbebaebd3_07d3_45d4_b85e_364a006910b2.slice/crio-eaac3265115a94b7adfe7e752396dae25a7a66d27888e29939e81c3f92155903 WatchSource:0}: Error finding container eaac3265115a94b7adfe7e752396dae25a7a66d27888e29939e81c3f92155903: Status 404 returned error can't find the container with id eaac3265115a94b7adfe7e752396dae25a7a66d27888e29939e81c3f92155903
Sep 11 18:27:37.657542 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.657511    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7647-dcdd43"]
Sep 11 18:27:37.663263 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.663235    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7647-dcdd43"]
Sep 11 18:27:37.775929 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.775848    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7647-3b809a"]
Sep 11 18:27:37.776303 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.776285    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a5c535-124b-44cc-bd62-820d6ee627e1" containerName="test-curl-7647-dcdd43"
Sep 11 18:27:37.776386 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.776305    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a5c535-124b-44cc-bd62-820d6ee627e1" containerName="test-curl-7647-dcdd43"
Sep 11 18:27:37.776439 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.776396    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7a5c535-124b-44cc-bd62-820d6ee627e1" containerName="test-curl-7647-dcdd43"
Sep 11 18:27:37.779338 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.779319    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-3b809a"
Sep 11 18:27:37.786626 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.786524    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7647-3b809a"]
Sep 11 18:27:37.899068 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.899024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhbtx\" (UniqueName: \"kubernetes.io/projected/977daca5-beb7-46d2-8727-828ead9aff60-kube-api-access-xhbtx\") pod \"test-curl-7647-3b809a\" (UID: \"977daca5-beb7-46d2-8727-828ead9aff60\") " pod="opendatahub/test-curl-7647-3b809a"
Sep 11 18:27:37.999917 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:37.999882    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhbtx\" (UniqueName: \"kubernetes.io/projected/977daca5-beb7-46d2-8727-828ead9aff60-kube-api-access-xhbtx\") pod \"test-curl-7647-3b809a\" (UID: \"977daca5-beb7-46d2-8727-828ead9aff60\") " pod="opendatahub/test-curl-7647-3b809a"
Sep 11 18:27:38.018771 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.018745    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhbtx\" (UniqueName: \"kubernetes.io/projected/977daca5-beb7-46d2-8727-828ead9aff60-kube-api-access-xhbtx\") pod \"test-curl-7647-3b809a\" (UID: \"977daca5-beb7-46d2-8727-828ead9aff60\") " pod="opendatahub/test-curl-7647-3b809a"
Sep 11 18:27:38.058306 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.058165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp" event={"ID":"bebaebd3-07d3-45d4-b85e-364a006910b2","Type":"ContainerStarted","Data":"ff629c35a750e74362054349a8eb0e75494e5182e0266f919bb4806c90a6199c"}
Sep 11 18:27:38.058306 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.058243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp" event={"ID":"bebaebd3-07d3-45d4-b85e-364a006910b2","Type":"ContainerStarted","Data":"eaac3265115a94b7adfe7e752396dae25a7a66d27888e29939e81c3f92155903"}
Sep 11 18:27:38.058306 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.058270    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"
Sep 11 18:27:38.060216 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.060188    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl" event={"ID":"12b432db-84df-4c1c-b999-a20494bb1663","Type":"ContainerStarted","Data":"8b507b233cdb0f5141e8e4b5d9929e37bd9d87f2a4c9b24e33f1b64cee74da1f"}
Sep 11 18:27:38.060366 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.060236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl" event={"ID":"12b432db-84df-4c1c-b999-a20494bb1663","Type":"ContainerStarted","Data":"6a7d79b0c0e7a908c05179a6121839a28ec400c59cc4a688a149ff2b28fec14b"}
Sep 11 18:27:38.060366 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:27:38.060284    2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7647-dcdd43" container="test-curl-7647-dcdd43"
Sep 11 18:27:38.077242 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.077177    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp" podStartSLOduration=2.07716164 podStartE2EDuration="2.07716164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:38.074795846 +0000 UTC m=+734.815922690" watchObservedRunningTime="2026-09-11 18:27:38.07716164 +0000 UTC m=+734.818288482"
Sep 11 18:27:38.088917 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.088892    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-3b809a"
Sep 11 18:27:38.090585 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.090548    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl" podStartSLOduration=2.090533087 podStartE2EDuration="2.090533087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:38.088882629 +0000 UTC m=+734.830009471" watchObservedRunningTime="2026-09-11 18:27:38.090533087 +0000 UTC m=+734.831659930"
Sep 11 18:27:38.137309 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.137274    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"]
Sep 11 18:27:38.145876 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.145836    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"]
Sep 11 18:27:38.146055 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.145974    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"
Sep 11 18:27:38.148237 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.148209    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7d7509\""
Sep 11 18:27:38.148371 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.148209    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7d7509-dockercfg-8gx2m\""
Sep 11 18:27:38.186548 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.186520    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"]
Sep 11 18:27:38.190529 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.190503    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"
Sep 11 18:27:38.197453 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.197429    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"]
Sep 11 18:27:38.303660 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.303625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhbds\" (UniqueName: \"kubernetes.io/projected/253be52b-26e8-4852-969d-de83b2166df7-kube-api-access-mhbds\") pod \"payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j\" (UID: \"253be52b-26e8-4852-969d-de83b2166df7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"
Sep 11 18:27:38.303801 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.303666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/253be52b-26e8-4852-969d-de83b2166df7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j\" (UID: \"253be52b-26e8-4852-969d-de83b2166df7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"
Sep 11 18:27:38.303855 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.303792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n\" (UID: \"6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"
Sep 11 18:27:38.303855 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.303822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dzw2\" (UniqueName: \"kubernetes.io/projected/6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f-kube-api-access-7dzw2\") pod \"payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n\" (UID: \"6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"
Sep 11 18:27:38.388397 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.388372    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7647-3b809a"]
Sep 11 18:27:38.390512 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:27:38.390467    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod977daca5_beb7_46d2_8727_828ead9aff60.slice/crio-9f0045cd2c8a792eedbcc3d91d08d0be593f77634eaca02b07fbfbd3a7812362 WatchSource:0}: Error finding container 9f0045cd2c8a792eedbcc3d91d08d0be593f77634eaca02b07fbfbd3a7812362: Status 404 returned error can't find the container with id 9f0045cd2c8a792eedbcc3d91d08d0be593f77634eaca02b07fbfbd3a7812362
Sep 11 18:27:38.404401 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.404377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n\" (UID: \"6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"
Sep 11 18:27:38.404541 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.404415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dzw2\" (UniqueName: \"kubernetes.io/projected/6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f-kube-api-access-7dzw2\") pod \"payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n\" (UID: \"6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"
Sep 11 18:27:38.404541 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.404472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhbds\" (UniqueName: \"kubernetes.io/projected/253be52b-26e8-4852-969d-de83b2166df7-kube-api-access-mhbds\") pod \"payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j\" (UID: \"253be52b-26e8-4852-969d-de83b2166df7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"
Sep 11 18:27:38.404655 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.404542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/253be52b-26e8-4852-969d-de83b2166df7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j\" (UID: \"253be52b-26e8-4852-969d-de83b2166df7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"
Sep 11 18:27:38.405092 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.405069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n\" (UID: \"6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"
Sep 11 18:27:38.405238 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.405217    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/253be52b-26e8-4852-969d-de83b2166df7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j\" (UID: \"253be52b-26e8-4852-969d-de83b2166df7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"
Sep 11 18:27:38.417827 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.417806    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dzw2\" (UniqueName: \"kubernetes.io/projected/6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f-kube-api-access-7dzw2\") pod \"payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n\" (UID: \"6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"
Sep 11 18:27:38.419399 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.419375    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhbds\" (UniqueName: \"kubernetes.io/projected/253be52b-26e8-4852-969d-de83b2166df7-kube-api-access-mhbds\") pod \"payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j\" (UID: \"253be52b-26e8-4852-969d-de83b2166df7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"
Sep 11 18:27:38.463165 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.463143    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"
Sep 11 18:27:38.504178 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.504146    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"
Sep 11 18:27:38.749054 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.749027    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"]
Sep 11 18:27:38.750827 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:27:38.750803    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253be52b_26e8_4852_969d_de83b2166df7.slice/crio-94dc87e93fe50e8b5306ca2f3eeb8e8aad338d531737cbd218f05152f615d3ff WatchSource:0}: Error finding container 94dc87e93fe50e8b5306ca2f3eeb8e8aad338d531737cbd218f05152f615d3ff: Status 404 returned error can't find the container with id 94dc87e93fe50e8b5306ca2f3eeb8e8aad338d531737cbd218f05152f615d3ff
Sep 11 18:27:38.798199 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:38.798178    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"]
Sep 11 18:27:38.799347 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:27:38.799328    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8fcdbc_67fa_4ef5_82f6_f3a60666fc7f.slice/crio-3963c5915f60bd657f4684f0d01bd47a4daf0b33162a73fd1fdeb1659af9af69 WatchSource:0}: Error finding container 3963c5915f60bd657f4684f0d01bd47a4daf0b33162a73fd1fdeb1659af9af69: Status 404 returned error can't find the container with id 3963c5915f60bd657f4684f0d01bd47a4daf0b33162a73fd1fdeb1659af9af69
Sep 11 18:27:39.067210 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:39.067127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j" event={"ID":"253be52b-26e8-4852-969d-de83b2166df7","Type":"ContainerStarted","Data":"4fe023e88ce1b55da5ba0f4324b2fb5f1b6d868c05c1572ab7f507b37747657a"}
Sep 11 18:27:39.067210 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:39.067191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j" event={"ID":"253be52b-26e8-4852-969d-de83b2166df7","Type":"ContainerStarted","Data":"94dc87e93fe50e8b5306ca2f3eeb8e8aad338d531737cbd218f05152f615d3ff"}
Sep 11 18:27:39.067674 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:39.067398    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"
Sep 11 18:27:39.069291 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:39.069246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7647-3b809a" event={"ID":"977daca5-beb7-46d2-8727-828ead9aff60","Type":"ContainerStarted","Data":"e03389f713369313ce027d8fa0ae8cb067fb972272038df6aa53cbce3ee6954c"}
Sep 11 18:27:39.069291 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:39.069285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7647-3b809a" event={"ID":"977daca5-beb7-46d2-8727-828ead9aff60","Type":"ContainerStarted","Data":"9f0045cd2c8a792eedbcc3d91d08d0be593f77634eaca02b07fbfbd3a7812362"}
Sep 11 18:27:39.071228 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:39.071118    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n" event={"ID":"6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f","Type":"ContainerStarted","Data":"6ab32ce74ccd2a440b56fd9d25a98d2de509ec5d837af4170259b4373614518b"}
Sep 11 18:27:39.071228 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:39.071176    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n" event={"ID":"6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f","Type":"ContainerStarted","Data":"3963c5915f60bd657f4684f0d01bd47a4daf0b33162a73fd1fdeb1659af9af69"}
Sep 11 18:27:39.071401 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:39.071213    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7647-dcdd43" podUID="e7a5c535-124b-44cc-bd62-820d6ee627e1" containerName="test-curl-7647-dcdd43" containerID="cri-o://7294aadad3cc47643bde6ecbe4f8fb6e4d2b1d638edbc1a3ab2126a3cf122a01" gracePeriod=2
Sep 11 18:27:39.071461 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:39.071448    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"
Sep 11 18:27:39.072088 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:39.071747    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"
Sep 11 18:27:39.084054 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:39.083985    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j" podStartSLOduration=1.083971906 podStartE2EDuration="1.083971906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:39.082107547 +0000 UTC m=+735.823234390" watchObservedRunningTime="2026-09-11 18:27:39.083971906 +0000 UTC m=+735.825098746"
Sep 11 18:27:39.096337 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:39.096296    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7647-3b809a" podStartSLOduration=2.096286415 podStartE2EDuration="2.096286415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:39.095164557 +0000 UTC m=+735.836291401" watchObservedRunningTime="2026-09-11 18:27:39.096286415 +0000 UTC m=+735.837413257"
Sep 11 18:27:39.111280 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:39.111063    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n" podStartSLOduration=1.111030452 podStartE2EDuration="1.111030452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:39.110860238 +0000 UTC m=+735.851987083" watchObservedRunningTime="2026-09-11 18:27:39.111030452 +0000 UTC m=+735.852157314"
Sep 11 18:27:39.683662 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:39.683609    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7647-3b809a"]
Sep 11 18:27:39.693527 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:39.693493    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7647-3b809a"]
Sep 11 18:27:40.077250 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:27:40.077213    2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7647-3b809a" container="test-curl-7647-3b809a"
Sep 11 18:27:41.082851 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:41.082820    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7647-3b809a" podUID="977daca5-beb7-46d2-8727-828ead9aff60" containerName="test-curl-7647-3b809a" containerID="cri-o://e03389f713369313ce027d8fa0ae8cb067fb972272038df6aa53cbce3ee6954c" gracePeriod=2
Sep 11 18:27:41.214548 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:41.214521    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-dcdd43"
Sep 11 18:27:41.333326 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:41.333249    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxx4t\" (UniqueName: \"kubernetes.io/projected/e7a5c535-124b-44cc-bd62-820d6ee627e1-kube-api-access-mxx4t\") pod \"e7a5c535-124b-44cc-bd62-820d6ee627e1\" (UID: \"e7a5c535-124b-44cc-bd62-820d6ee627e1\") "
Sep 11 18:27:41.335526 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:41.335503    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a5c535-124b-44cc-bd62-820d6ee627e1-kube-api-access-mxx4t" (OuterVolumeSpecName: "kube-api-access-mxx4t") pod "e7a5c535-124b-44cc-bd62-820d6ee627e1" (UID: "e7a5c535-124b-44cc-bd62-820d6ee627e1"). InnerVolumeSpecName "kube-api-access-mxx4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:27:41.434275 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:41.434248    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxx4t\" (UniqueName: \"kubernetes.io/projected/e7a5c535-124b-44cc-bd62-820d6ee627e1-kube-api-access-mxx4t\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:27:41.763902 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:41.763862    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7a5c535-124b-44cc-bd62-820d6ee627e1" path="/var/lib/kubelet/pods/e7a5c535-124b-44cc-bd62-820d6ee627e1/volumes"
Sep 11 18:27:42.097225 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:42.097137    2581 generic.go:356] "Generic (PLEG): container finished" podID="e7a5c535-124b-44cc-bd62-820d6ee627e1" containerID="7294aadad3cc47643bde6ecbe4f8fb6e4d2b1d638edbc1a3ab2126a3cf122a01" exitCode=137
Sep 11 18:27:42.097225 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:42.097189    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-dcdd43"
Sep 11 18:27:42.097675 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:42.097241    2581 scope.go:117] "RemoveContainer" containerID="7294aadad3cc47643bde6ecbe4f8fb6e4d2b1d638edbc1a3ab2126a3cf122a01"
Sep 11 18:27:42.105690 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:42.105672    2581 scope.go:117] "RemoveContainer" containerID="7294aadad3cc47643bde6ecbe4f8fb6e4d2b1d638edbc1a3ab2126a3cf122a01"
Sep 11 18:27:42.106072 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:27:42.106042    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7294aadad3cc47643bde6ecbe4f8fb6e4d2b1d638edbc1a3ab2126a3cf122a01\": container with ID starting with 7294aadad3cc47643bde6ecbe4f8fb6e4d2b1d638edbc1a3ab2126a3cf122a01 not found: ID does not exist" containerID="7294aadad3cc47643bde6ecbe4f8fb6e4d2b1d638edbc1a3ab2126a3cf122a01"
Sep 11 18:27:42.106136 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:42.106083    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7294aadad3cc47643bde6ecbe4f8fb6e4d2b1d638edbc1a3ab2126a3cf122a01"} err="failed to get container status \"7294aadad3cc47643bde6ecbe4f8fb6e4d2b1d638edbc1a3ab2126a3cf122a01\": rpc error: code = NotFound desc = could not find container \"7294aadad3cc47643bde6ecbe4f8fb6e4d2b1d638edbc1a3ab2126a3cf122a01\": container with ID starting with 7294aadad3cc47643bde6ecbe4f8fb6e4d2b1d638edbc1a3ab2126a3cf122a01 not found: ID does not exist"
Sep 11 18:27:42.873937 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:42.873910    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"]
Sep 11 18:27:42.874374 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:42.874361    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977daca5-beb7-46d2-8727-828ead9aff60" containerName="test-curl-7647-3b809a"
Sep 11 18:27:42.874420 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:42.874376    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="977daca5-beb7-46d2-8727-828ead9aff60" containerName="test-curl-7647-3b809a"
Sep 11 18:27:42.874494 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:42.874470    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="977daca5-beb7-46d2-8727-828ead9aff60" containerName="test-curl-7647-3b809a"
Sep 11 18:27:42.879006 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:42.878991    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"
Sep 11 18:27:42.881029 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:42.881004    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-724327\""
Sep 11 18:27:42.881029 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:42.881021    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-724327-dockercfg-kf4w4\""
Sep 11 18:27:42.888941 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:42.888917    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"]
Sep 11 18:27:42.933111 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:42.933086    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"]
Sep 11 18:27:42.936440 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:42.936426    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"
Sep 11 18:27:42.946080 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:42.946061    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"]
Sep 11 18:27:43.049932 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.049903    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpbxq\" (UniqueName: \"kubernetes.io/projected/a8de682d-257c-4068-83a9-b3cb657ecddb-kube-api-access-dpbxq\") pod \"payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9\" (UID: \"a8de682d-257c-4068-83a9-b3cb657ecddb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"
Sep 11 18:27:43.050086 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.049940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8de682d-257c-4068-83a9-b3cb657ecddb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9\" (UID: \"a8de682d-257c-4068-83a9-b3cb657ecddb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"
Sep 11 18:27:43.050086 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.049973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5jp6\" (UniqueName: \"kubernetes.io/projected/f42b5999-d24c-46a2-9b2e-596e4680bcb8-kube-api-access-s5jp6\") pod \"payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj\" (UID: \"f42b5999-d24c-46a2-9b2e-596e4680bcb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"
Sep 11 18:27:43.050202 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.050134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42b5999-d24c-46a2-9b2e-596e4680bcb8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj\" (UID: \"f42b5999-d24c-46a2-9b2e-596e4680bcb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"
Sep 11 18:27:43.150673 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.150639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42b5999-d24c-46a2-9b2e-596e4680bcb8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj\" (UID: \"f42b5999-d24c-46a2-9b2e-596e4680bcb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"
Sep 11 18:27:43.151063 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.150702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpbxq\" (UniqueName: \"kubernetes.io/projected/a8de682d-257c-4068-83a9-b3cb657ecddb-kube-api-access-dpbxq\") pod \"payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9\" (UID: \"a8de682d-257c-4068-83a9-b3cb657ecddb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"
Sep 11 18:27:43.151063 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.150737    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8de682d-257c-4068-83a9-b3cb657ecddb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9\" (UID: \"a8de682d-257c-4068-83a9-b3cb657ecddb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"
Sep 11 18:27:43.151063 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.150879    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5jp6\" (UniqueName: \"kubernetes.io/projected/f42b5999-d24c-46a2-9b2e-596e4680bcb8-kube-api-access-s5jp6\") pod \"payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj\" (UID: \"f42b5999-d24c-46a2-9b2e-596e4680bcb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"
Sep 11 18:27:43.151579 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.151557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8de682d-257c-4068-83a9-b3cb657ecddb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9\" (UID: \"a8de682d-257c-4068-83a9-b3cb657ecddb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"
Sep 11 18:27:43.151675 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.151557    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42b5999-d24c-46a2-9b2e-596e4680bcb8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj\" (UID: \"f42b5999-d24c-46a2-9b2e-596e4680bcb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"
Sep 11 18:27:43.159785 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.159764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpbxq\" (UniqueName: \"kubernetes.io/projected/a8de682d-257c-4068-83a9-b3cb657ecddb-kube-api-access-dpbxq\") pod \"payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9\" (UID: \"a8de682d-257c-4068-83a9-b3cb657ecddb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"
Sep 11 18:27:43.159907 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.159849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5jp6\" (UniqueName: \"kubernetes.io/projected/f42b5999-d24c-46a2-9b2e-596e4680bcb8-kube-api-access-s5jp6\") pod \"payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj\" (UID: \"f42b5999-d24c-46a2-9b2e-596e4680bcb8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"
Sep 11 18:27:43.190142 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.190105    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"
Sep 11 18:27:43.246228 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.246202    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"
Sep 11 18:27:43.250031 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.250006    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-3b809a"
Sep 11 18:27:43.353183 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.353153    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhbtx\" (UniqueName: \"kubernetes.io/projected/977daca5-beb7-46d2-8727-828ead9aff60-kube-api-access-xhbtx\") pod \"977daca5-beb7-46d2-8727-828ead9aff60\" (UID: \"977daca5-beb7-46d2-8727-828ead9aff60\") "
Sep 11 18:27:43.355268 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.355241    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977daca5-beb7-46d2-8727-828ead9aff60-kube-api-access-xhbtx" (OuterVolumeSpecName: "kube-api-access-xhbtx") pod "977daca5-beb7-46d2-8727-828ead9aff60" (UID: "977daca5-beb7-46d2-8727-828ead9aff60"). InnerVolumeSpecName "kube-api-access-xhbtx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:27:43.454352 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.454288    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhbtx\" (UniqueName: \"kubernetes.io/projected/977daca5-beb7-46d2-8727-828ead9aff60-kube-api-access-xhbtx\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:27:43.471708 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.471684    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"]
Sep 11 18:27:43.473647 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:27:43.473625    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42b5999_d24c_46a2_9b2e_596e4680bcb8.slice/crio-191721a19b848127bf0d2bb58eab2de3e3f5d2ed52b7043822845ef0831bc3ca WatchSource:0}: Error finding container 191721a19b848127bf0d2bb58eab2de3e3f5d2ed52b7043822845ef0831bc3ca: Status 404 returned error can't find the container with id 191721a19b848127bf0d2bb58eab2de3e3f5d2ed52b7043822845ef0831bc3ca
Sep 11 18:27:43.525701 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.525650    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"]
Sep 11 18:27:43.527827 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:27:43.527799    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8de682d_257c_4068_83a9_b3cb657ecddb.slice/crio-7ebc82cea3402952e2cbbae41c64805b8e321b6f4f8eb76834f7fd315fc8059b WatchSource:0}: Error finding container 7ebc82cea3402952e2cbbae41c64805b8e321b6f4f8eb76834f7fd315fc8059b: Status 404 returned error can't find the container with id 7ebc82cea3402952e2cbbae41c64805b8e321b6f4f8eb76834f7fd315fc8059b
Sep 11 18:27:43.765653 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:43.765581    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="977daca5-beb7-46d2-8727-828ead9aff60" path="/var/lib/kubelet/pods/977daca5-beb7-46d2-8727-828ead9aff60/volumes"
Sep 11 18:27:44.110623 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:44.110595    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj" event={"ID":"f42b5999-d24c-46a2-9b2e-596e4680bcb8","Type":"ContainerStarted","Data":"36daa507a65d65ddd62d3e2ec1723636e532f91c5881b1697861b3614f6db5f3"}
Sep 11 18:27:44.110762 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:44.110643    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj" event={"ID":"f42b5999-d24c-46a2-9b2e-596e4680bcb8","Type":"ContainerStarted","Data":"191721a19b848127bf0d2bb58eab2de3e3f5d2ed52b7043822845ef0831bc3ca"}
Sep 11 18:27:44.110762 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:44.110703    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"
Sep 11 18:27:44.112689 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:44.112667    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9" event={"ID":"a8de682d-257c-4068-83a9-b3cb657ecddb","Type":"ContainerStarted","Data":"7db7e900d28aed8c895395748ba1e8bec29d718ac2c16d5f835fd8678fa00e9a"}
Sep 11 18:27:44.112799 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:44.112709    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9" event={"ID":"a8de682d-257c-4068-83a9-b3cb657ecddb","Type":"ContainerStarted","Data":"7ebc82cea3402952e2cbbae41c64805b8e321b6f4f8eb76834f7fd315fc8059b"}
Sep 11 18:27:44.112799 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:44.112757    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"
Sep 11 18:27:44.114048 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:44.114024    2581 generic.go:356] "Generic (PLEG): container finished" podID="977daca5-beb7-46d2-8727-828ead9aff60" containerID="e03389f713369313ce027d8fa0ae8cb067fb972272038df6aa53cbce3ee6954c" exitCode=137
Sep 11 18:27:44.114136 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:44.114070    2581 scope.go:117] "RemoveContainer" containerID="e03389f713369313ce027d8fa0ae8cb067fb972272038df6aa53cbce3ee6954c"
Sep 11 18:27:44.114136 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:44.114062    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7647-3b809a"
Sep 11 18:27:44.130870 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:44.130812    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj" podStartSLOduration=2.130799365 podStartE2EDuration="2.130799365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:44.129205676 +0000 UTC m=+740.870332530" watchObservedRunningTime="2026-09-11 18:27:44.130799365 +0000 UTC m=+740.871926208"
Sep 11 18:27:44.132820 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:44.132798    2581 scope.go:117] "RemoveContainer" containerID="e03389f713369313ce027d8fa0ae8cb067fb972272038df6aa53cbce3ee6954c"
Sep 11 18:27:44.133065 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:27:44.133047    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e03389f713369313ce027d8fa0ae8cb067fb972272038df6aa53cbce3ee6954c\": container with ID starting with e03389f713369313ce027d8fa0ae8cb067fb972272038df6aa53cbce3ee6954c not found: ID does not exist" containerID="e03389f713369313ce027d8fa0ae8cb067fb972272038df6aa53cbce3ee6954c"
Sep 11 18:27:44.133126 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:44.133072    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e03389f713369313ce027d8fa0ae8cb067fb972272038df6aa53cbce3ee6954c"} err="failed to get container status \"e03389f713369313ce027d8fa0ae8cb067fb972272038df6aa53cbce3ee6954c\": rpc error: code = NotFound desc = could not find container \"e03389f713369313ce027d8fa0ae8cb067fb972272038df6aa53cbce3ee6954c\": container with ID starting with e03389f713369313ce027d8fa0ae8cb067fb972272038df6aa53cbce3ee6954c not found: ID does not exist"
Sep 11 18:27:44.149250 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:44.149206    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9" podStartSLOduration=2.149190393 podStartE2EDuration="2.149190393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:44.14831442 +0000 UTC m=+740.889441289" watchObservedRunningTime="2026-09-11 18:27:44.149190393 +0000 UTC m=+740.890317237"
Sep 11 18:27:47.156559 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.156525    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"]
Sep 11 18:27:47.161652 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.161632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"
Sep 11 18:27:47.163742 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.163722    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-7e8ab9-dockercfg-s5p59\""
Sep 11 18:27:47.163844 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.163728    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-7e8ab9\""
Sep 11 18:27:47.168788 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.168770    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"]
Sep 11 18:27:47.172921 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.172897    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"]
Sep 11 18:27:47.173010 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.172973    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"
Sep 11 18:27:47.185340 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.185312    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"]
Sep 11 18:27:47.290049 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.290014    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flfn6\" (UniqueName: \"kubernetes.io/projected/ad2e2354-3592-4381-b6e4-f80cc20bcb1f-kube-api-access-flfn6\") pod \"payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5\" (UID: \"ad2e2354-3592-4381-b6e4-f80cc20bcb1f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"
Sep 11 18:27:47.290208 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.290080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a69185-1e46-45ec-9b16-448dfa377375-plugins-config-volume\") pod \"payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7\" (UID: \"74a69185-1e46-45ec-9b16-448dfa377375\") " pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"
Sep 11 18:27:47.290208 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.290104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2e2354-3592-4381-b6e4-f80cc20bcb1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5\" (UID: \"ad2e2354-3592-4381-b6e4-f80cc20bcb1f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"
Sep 11 18:27:47.290208 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.290155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsb5g\" (UniqueName: \"kubernetes.io/projected/74a69185-1e46-45ec-9b16-448dfa377375-kube-api-access-wsb5g\") pod \"payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7\" (UID: \"74a69185-1e46-45ec-9b16-448dfa377375\") " pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"
Sep 11 18:27:47.391338 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.391307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsb5g\" (UniqueName: \"kubernetes.io/projected/74a69185-1e46-45ec-9b16-448dfa377375-kube-api-access-wsb5g\") pod \"payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7\" (UID: \"74a69185-1e46-45ec-9b16-448dfa377375\") " pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"
Sep 11 18:27:47.391576 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.391391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flfn6\" (UniqueName: \"kubernetes.io/projected/ad2e2354-3592-4381-b6e4-f80cc20bcb1f-kube-api-access-flfn6\") pod \"payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5\" (UID: \"ad2e2354-3592-4381-b6e4-f80cc20bcb1f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"
Sep 11 18:27:47.391576 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.391434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a69185-1e46-45ec-9b16-448dfa377375-plugins-config-volume\") pod \"payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7\" (UID: \"74a69185-1e46-45ec-9b16-448dfa377375\") " pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"
Sep 11 18:27:47.391576 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.391450    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2e2354-3592-4381-b6e4-f80cc20bcb1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5\" (UID: \"ad2e2354-3592-4381-b6e4-f80cc20bcb1f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"
Sep 11 18:27:47.392358 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.392336    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a69185-1e46-45ec-9b16-448dfa377375-plugins-config-volume\") pod \"payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7\" (UID: \"74a69185-1e46-45ec-9b16-448dfa377375\") " pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"
Sep 11 18:27:47.392402 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.392336    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2e2354-3592-4381-b6e4-f80cc20bcb1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5\" (UID: \"ad2e2354-3592-4381-b6e4-f80cc20bcb1f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"
Sep 11 18:27:47.400336 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.400304    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsb5g\" (UniqueName: \"kubernetes.io/projected/74a69185-1e46-45ec-9b16-448dfa377375-kube-api-access-wsb5g\") pod \"payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7\" (UID: \"74a69185-1e46-45ec-9b16-448dfa377375\") " pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"
Sep 11 18:27:47.400631 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.400609    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flfn6\" (UniqueName: \"kubernetes.io/projected/ad2e2354-3592-4381-b6e4-f80cc20bcb1f-kube-api-access-flfn6\") pod \"payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5\" (UID: \"ad2e2354-3592-4381-b6e4-f80cc20bcb1f\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"
Sep 11 18:27:47.474327 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.474257    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"
Sep 11 18:27:47.484737 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.484714    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"
Sep 11 18:27:47.761876 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:27:47.761849    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2e2354_3592_4381_b6e4_f80cc20bcb1f.slice/crio-8b7f338390239a1cc28c45a843b62c4957cdb1936fd21b2a90c76e6a5b14fd09 WatchSource:0}: Error finding container 8b7f338390239a1cc28c45a843b62c4957cdb1936fd21b2a90c76e6a5b14fd09: Status 404 returned error can't find the container with id 8b7f338390239a1cc28c45a843b62c4957cdb1936fd21b2a90c76e6a5b14fd09
Sep 11 18:27:47.767201 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.767180    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"]
Sep 11 18:27:47.776919 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:47.776897    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"]
Sep 11 18:27:47.778835 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:27:47.778814    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a69185_1e46_45ec_9b16_448dfa377375.slice/crio-c698940afa18c8b72cdfad428323de18e095f68e31135fe63032c81cf45aafa2 WatchSource:0}: Error finding container c698940afa18c8b72cdfad428323de18e095f68e31135fe63032c81cf45aafa2: Status 404 returned error can't find the container with id c698940afa18c8b72cdfad428323de18e095f68e31135fe63032c81cf45aafa2
Sep 11 18:27:48.141183 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:48.141143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7" event={"ID":"74a69185-1e46-45ec-9b16-448dfa377375","Type":"ContainerStarted","Data":"5cb4ff45f24f605a47740192197c85e84b6c00f6e190da1c3ecd69997ed3cd7a"}
Sep 11 18:27:48.141340 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:48.141207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7" event={"ID":"74a69185-1e46-45ec-9b16-448dfa377375","Type":"ContainerStarted","Data":"c698940afa18c8b72cdfad428323de18e095f68e31135fe63032c81cf45aafa2"}
Sep 11 18:27:48.141418 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:48.141349    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"
Sep 11 18:27:48.143142 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:48.143111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5" event={"ID":"ad2e2354-3592-4381-b6e4-f80cc20bcb1f","Type":"ContainerStarted","Data":"1ad86bcb997e4049d286697f4b7d00f38e2c30f3178d0ce8833945ac515616e3"}
Sep 11 18:27:48.143270 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:48.143157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5" event={"ID":"ad2e2354-3592-4381-b6e4-f80cc20bcb1f","Type":"ContainerStarted","Data":"8b7f338390239a1cc28c45a843b62c4957cdb1936fd21b2a90c76e6a5b14fd09"}
Sep 11 18:27:48.143270 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:48.143185    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"
Sep 11 18:27:48.162054 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:48.161995    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7" podStartSLOduration=1.161975846 podStartE2EDuration="1.161975846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:48.160637658 +0000 UTC m=+744.901764503" watchObservedRunningTime="2026-09-11 18:27:48.161975846 +0000 UTC m=+744.903102690"
Sep 11 18:27:48.181969 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:48.181924    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5" podStartSLOduration=1.18191188 podStartE2EDuration="1.18191188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:48.179344326 +0000 UTC m=+744.920471170" watchObservedRunningTime="2026-09-11 18:27:48.18191188 +0000 UTC m=+744.923038723"
Sep 11 18:27:48.960575 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:48.960537    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"]
Sep 11 18:27:48.964312 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:48.964295    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"
Sep 11 18:27:48.966382 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:48.966359    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8370956b\""
Sep 11 18:27:48.975745 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:48.975704    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"]
Sep 11 18:27:48.979422 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:48.979399    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"]
Sep 11 18:27:48.983597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:48.983576    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"
Sep 11 18:27:48.986520 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:48.985936    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8370956b-dockercfg-tv78s\""
Sep 11 18:27:48.994848 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:48.994826    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"]
Sep 11 18:27:49.073019 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.072992    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"
Sep 11 18:27:49.107099 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.107073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f695a9e-3528-41e7-a63d-969bebd396ac-plugins-config-volume\") pod \"payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc\" (UID: \"6f695a9e-3528-41e7-a63d-969bebd396ac\") " pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"
Sep 11 18:27:49.107260 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.107115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lbg5\" (UniqueName: \"kubernetes.io/projected/6f695a9e-3528-41e7-a63d-969bebd396ac-kube-api-access-4lbg5\") pod \"payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc\" (UID: \"6f695a9e-3528-41e7-a63d-969bebd396ac\") " pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"
Sep 11 18:27:49.107260 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.107199    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtvbf\" (UniqueName: \"kubernetes.io/projected/25769e2e-02ea-4c99-bdc9-7278c6b5fc67-kube-api-access-jtvbf\") pod \"payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k\" (UID: \"25769e2e-02ea-4c99-bdc9-7278c6b5fc67\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"
Sep 11 18:27:49.107370 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.107321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25769e2e-02ea-4c99-bdc9-7278c6b5fc67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k\" (UID: \"25769e2e-02ea-4c99-bdc9-7278c6b5fc67\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"
Sep 11 18:27:49.208563 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.208528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f695a9e-3528-41e7-a63d-969bebd396ac-plugins-config-volume\") pod \"payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc\" (UID: \"6f695a9e-3528-41e7-a63d-969bebd396ac\") " pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"
Sep 11 18:27:49.208986 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.208571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lbg5\" (UniqueName: \"kubernetes.io/projected/6f695a9e-3528-41e7-a63d-969bebd396ac-kube-api-access-4lbg5\") pod \"payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc\" (UID: \"6f695a9e-3528-41e7-a63d-969bebd396ac\") " pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"
Sep 11 18:27:49.208986 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.208623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtvbf\" (UniqueName: \"kubernetes.io/projected/25769e2e-02ea-4c99-bdc9-7278c6b5fc67-kube-api-access-jtvbf\") pod \"payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k\" (UID: \"25769e2e-02ea-4c99-bdc9-7278c6b5fc67\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"
Sep 11 18:27:49.208986 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.208735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25769e2e-02ea-4c99-bdc9-7278c6b5fc67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k\" (UID: \"25769e2e-02ea-4c99-bdc9-7278c6b5fc67\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"
Sep 11 18:27:49.209394 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.209371    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f695a9e-3528-41e7-a63d-969bebd396ac-plugins-config-volume\") pod \"payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc\" (UID: \"6f695a9e-3528-41e7-a63d-969bebd396ac\") " pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"
Sep 11 18:27:49.209394 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.209387    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25769e2e-02ea-4c99-bdc9-7278c6b5fc67-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k\" (UID: \"25769e2e-02ea-4c99-bdc9-7278c6b5fc67\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"
Sep 11 18:27:49.217929 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.217868    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lbg5\" (UniqueName: \"kubernetes.io/projected/6f695a9e-3528-41e7-a63d-969bebd396ac-kube-api-access-4lbg5\") pod \"payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc\" (UID: \"6f695a9e-3528-41e7-a63d-969bebd396ac\") " pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"
Sep 11 18:27:49.218038 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.217970    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtvbf\" (UniqueName: \"kubernetes.io/projected/25769e2e-02ea-4c99-bdc9-7278c6b5fc67-kube-api-access-jtvbf\") pod \"payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k\" (UID: \"25769e2e-02ea-4c99-bdc9-7278c6b5fc67\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"
Sep 11 18:27:49.277747 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.277719    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"
Sep 11 18:27:49.295163 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.295140    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"
Sep 11 18:27:49.575906 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.575881    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"]
Sep 11 18:27:49.577855 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:27:49.577821    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25769e2e_02ea_4c99_bdc9_7278c6b5fc67.slice/crio-e772680c4f3d664e605fd787c3ee2601039dc6149a72406d4d0348ca005a7f60 WatchSource:0}: Error finding container e772680c4f3d664e605fd787c3ee2601039dc6149a72406d4d0348ca005a7f60: Status 404 returned error can't find the container with id e772680c4f3d664e605fd787c3ee2601039dc6149a72406d4d0348ca005a7f60
Sep 11 18:27:49.583134 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:49.583108    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"]
Sep 11 18:27:49.584623 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:27:49.584600    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f695a9e_3528_41e7_a63d_969bebd396ac.slice/crio-26425d355aa2a1cd7f5f3687d1d220d77a376a16980a3670ca1b35e85a4a155f WatchSource:0}: Error finding container 26425d355aa2a1cd7f5f3687d1d220d77a376a16980a3670ca1b35e85a4a155f: Status 404 returned error can't find the container with id 26425d355aa2a1cd7f5f3687d1d220d77a376a16980a3670ca1b35e85a4a155f
Sep 11 18:27:50.079643 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:50.079607    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"
Sep 11 18:27:50.080662 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:50.080634    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"
Sep 11 18:27:50.080764 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:50.080714    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"
Sep 11 18:27:50.162932 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:50.162898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc" event={"ID":"6f695a9e-3528-41e7-a63d-969bebd396ac","Type":"ContainerStarted","Data":"5c029e3179a020be9b0c3d895e6ab590e4827eb21707d003d129b808c46a535e"}
Sep 11 18:27:50.163159 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:50.163142    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc" event={"ID":"6f695a9e-3528-41e7-a63d-969bebd396ac","Type":"ContainerStarted","Data":"26425d355aa2a1cd7f5f3687d1d220d77a376a16980a3670ca1b35e85a4a155f"}
Sep 11 18:27:50.164208 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:50.164017    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"
Sep 11 18:27:50.166233 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:50.166211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k" event={"ID":"25769e2e-02ea-4c99-bdc9-7278c6b5fc67","Type":"ContainerStarted","Data":"a82fa9e8051da366ddbfde93ffb7af0d42895541d054576c50f4c2927a597a70"}
Sep 11 18:27:50.166341 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:50.166268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k" event={"ID":"25769e2e-02ea-4c99-bdc9-7278c6b5fc67","Type":"ContainerStarted","Data":"e772680c4f3d664e605fd787c3ee2601039dc6149a72406d4d0348ca005a7f60"}
Sep 11 18:27:50.166628 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:50.166608    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"
Sep 11 18:27:50.182650 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:50.182607    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc" podStartSLOduration=2.182591611 podStartE2EDuration="2.182591611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:50.180971704 +0000 UTC m=+746.922098548" watchObservedRunningTime="2026-09-11 18:27:50.182591611 +0000 UTC m=+746.923718453"
Sep 11 18:27:50.197052 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:50.197011    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k" podStartSLOduration=2.196997973 podStartE2EDuration="2.196997973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:50.196385139 +0000 UTC m=+746.937512025" watchObservedRunningTime="2026-09-11 18:27:50.196997973 +0000 UTC m=+746.938124815"
Sep 11 18:27:55.125065 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:55.125035    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"
Sep 11 18:27:55.125467 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:55.125101    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"
Sep 11 18:27:55.982462 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:55.982428    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:27:59.156542 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:59.156516    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"
Sep 11 18:27:59.156968 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:59.156590    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"
Sep 11 18:27:59.647787 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:59.647753    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"]
Sep 11 18:27:59.652676 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:59.652655    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"
Sep 11 18:27:59.654620 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:59.654597    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1de19d\""
Sep 11 18:27:59.654734 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:59.654637    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1de19d-dockercfg-bg8bv\""
Sep 11 18:27:59.662101 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:59.662079    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"]
Sep 11 18:27:59.705578 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:59.705548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3126d145-7b41-4488-8473-2d83263a3a52-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw\" (UID: \"3126d145-7b41-4488-8473-2d83263a3a52\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"
Sep 11 18:27:59.705712 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:59.705589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz9tv\" (UniqueName: \"kubernetes.io/projected/3126d145-7b41-4488-8473-2d83263a3a52-kube-api-access-pz9tv\") pod \"payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw\" (UID: \"3126d145-7b41-4488-8473-2d83263a3a52\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"
Sep 11 18:27:59.807071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:59.807044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3126d145-7b41-4488-8473-2d83263a3a52-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw\" (UID: \"3126d145-7b41-4488-8473-2d83263a3a52\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"
Sep 11 18:27:59.807212 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:59.807079    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz9tv\" (UniqueName: \"kubernetes.io/projected/3126d145-7b41-4488-8473-2d83263a3a52-kube-api-access-pz9tv\") pod \"payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw\" (UID: \"3126d145-7b41-4488-8473-2d83263a3a52\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"
Sep 11 18:27:59.807794 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:59.807772    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3126d145-7b41-4488-8473-2d83263a3a52-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw\" (UID: \"3126d145-7b41-4488-8473-2d83263a3a52\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"
Sep 11 18:27:59.817422 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:59.817401    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz9tv\" (UniqueName: \"kubernetes.io/projected/3126d145-7b41-4488-8473-2d83263a3a52-kube-api-access-pz9tv\") pod \"payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw\" (UID: \"3126d145-7b41-4488-8473-2d83263a3a52\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"
Sep 11 18:27:59.964209 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:27:59.964125    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"
Sep 11 18:28:00.237472 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:00.237447    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"]
Sep 11 18:28:00.239112 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:28:00.239088    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3126d145_7b41_4488_8473_2d83263a3a52.slice/crio-192d242ac5c7948bde56f3d29cf3b5cf55f967c9fd93b63e3a6b347c1dbd256a WatchSource:0}: Error finding container 192d242ac5c7948bde56f3d29cf3b5cf55f967c9fd93b63e3a6b347c1dbd256a: Status 404 returned error can't find the container with id 192d242ac5c7948bde56f3d29cf3b5cf55f967c9fd93b63e3a6b347c1dbd256a
Sep 11 18:28:01.223491 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:01.223448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw" event={"ID":"3126d145-7b41-4488-8473-2d83263a3a52","Type":"ContainerStarted","Data":"64b5beb4e7dd035e9a136ad0475b69f84365e168973332912276af258dd331a5"}
Sep 11 18:28:01.223641 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:01.223511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw" event={"ID":"3126d145-7b41-4488-8473-2d83263a3a52","Type":"ContainerStarted","Data":"192d242ac5c7948bde56f3d29cf3b5cf55f967c9fd93b63e3a6b347c1dbd256a"}
Sep 11 18:28:01.223641 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:01.223617    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"
Sep 11 18:28:01.239541 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:01.239467    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw" podStartSLOduration=2.239448979 podStartE2EDuration="2.239448979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:01.238950238 +0000 UTC m=+757.980077105" watchObservedRunningTime="2026-09-11 18:28:01.239448979 +0000 UTC m=+757.980575824"
Sep 11 18:28:02.177498 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:02.177452    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"
Sep 11 18:28:02.177677 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:02.177662    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"
Sep 11 18:28:12.231962 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:12.231931    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"
Sep 11 18:28:14.420216 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.420182    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"]
Sep 11 18:28:14.420681 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.420519    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc" podUID="6f695a9e-3528-41e7-a63d-969bebd396ac" containerName="payload-processing" containerID="cri-o://5c029e3179a020be9b0c3d895e6ab590e4827eb21707d003d129b808c46a535e" gracePeriod=30
Sep 11 18:28:14.429567 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.429531    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"]
Sep 11 18:28:14.429809 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.429785    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k" podUID="25769e2e-02ea-4c99-bdc9-7278c6b5fc67" containerName="payload-pre-processing" containerID="cri-o://a82fa9e8051da366ddbfde93ffb7af0d42895541d054576c50f4c2927a597a70" gracePeriod=30
Sep 11 18:28:14.705447 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.705428    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"
Sep 11 18:28:14.708625 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.708608    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"
Sep 11 18:28:14.846171 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.846140    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lbg5\" (UniqueName: \"kubernetes.io/projected/6f695a9e-3528-41e7-a63d-969bebd396ac-kube-api-access-4lbg5\") pod \"6f695a9e-3528-41e7-a63d-969bebd396ac\" (UID: \"6f695a9e-3528-41e7-a63d-969bebd396ac\") "
Sep 11 18:28:14.846347 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.846188    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f695a9e-3528-41e7-a63d-969bebd396ac-plugins-config-volume\") pod \"6f695a9e-3528-41e7-a63d-969bebd396ac\" (UID: \"6f695a9e-3528-41e7-a63d-969bebd396ac\") "
Sep 11 18:28:14.846347 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.846232    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25769e2e-02ea-4c99-bdc9-7278c6b5fc67-plugins-config-volume\") pod \"25769e2e-02ea-4c99-bdc9-7278c6b5fc67\" (UID: \"25769e2e-02ea-4c99-bdc9-7278c6b5fc67\") "
Sep 11 18:28:14.846347 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.846255    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtvbf\" (UniqueName: \"kubernetes.io/projected/25769e2e-02ea-4c99-bdc9-7278c6b5fc67-kube-api-access-jtvbf\") pod \"25769e2e-02ea-4c99-bdc9-7278c6b5fc67\" (UID: \"25769e2e-02ea-4c99-bdc9-7278c6b5fc67\") "
Sep 11 18:28:14.846651 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.846613    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f695a9e-3528-41e7-a63d-969bebd396ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f695a9e-3528-41e7-a63d-969bebd396ac" (UID: "6f695a9e-3528-41e7-a63d-969bebd396ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:28:14.846789 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.846648    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25769e2e-02ea-4c99-bdc9-7278c6b5fc67-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25769e2e-02ea-4c99-bdc9-7278c6b5fc67" (UID: "25769e2e-02ea-4c99-bdc9-7278c6b5fc67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:28:14.848454 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.848435    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f695a9e-3528-41e7-a63d-969bebd396ac-kube-api-access-4lbg5" (OuterVolumeSpecName: "kube-api-access-4lbg5") pod "6f695a9e-3528-41e7-a63d-969bebd396ac" (UID: "6f695a9e-3528-41e7-a63d-969bebd396ac"). InnerVolumeSpecName "kube-api-access-4lbg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:28:14.848551 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.848527    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25769e2e-02ea-4c99-bdc9-7278c6b5fc67-kube-api-access-jtvbf" (OuterVolumeSpecName: "kube-api-access-jtvbf") pod "25769e2e-02ea-4c99-bdc9-7278c6b5fc67" (UID: "25769e2e-02ea-4c99-bdc9-7278c6b5fc67"). InnerVolumeSpecName "kube-api-access-jtvbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:28:14.947817 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.947751    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lbg5\" (UniqueName: \"kubernetes.io/projected/6f695a9e-3528-41e7-a63d-969bebd396ac-kube-api-access-4lbg5\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:28:14.947817 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.947788    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f695a9e-3528-41e7-a63d-969bebd396ac-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:28:14.947817 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.947802    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25769e2e-02ea-4c99-bdc9-7278c6b5fc67-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:28:14.947817 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:14.947817    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtvbf\" (UniqueName: \"kubernetes.io/projected/25769e2e-02ea-4c99-bdc9-7278c6b5fc67-kube-api-access-jtvbf\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:28:15.299639 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.299553    2581 generic.go:356] "Generic (PLEG): container finished" podID="25769e2e-02ea-4c99-bdc9-7278c6b5fc67" containerID="a82fa9e8051da366ddbfde93ffb7af0d42895541d054576c50f4c2927a597a70" exitCode=0
Sep 11 18:28:15.299639 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.299601    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"
Sep 11 18:28:15.299835 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.299641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k" event={"ID":"25769e2e-02ea-4c99-bdc9-7278c6b5fc67","Type":"ContainerDied","Data":"a82fa9e8051da366ddbfde93ffb7af0d42895541d054576c50f4c2927a597a70"}
Sep 11 18:28:15.299835 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.299691    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k" event={"ID":"25769e2e-02ea-4c99-bdc9-7278c6b5fc67","Type":"ContainerDied","Data":"e772680c4f3d664e605fd787c3ee2601039dc6149a72406d4d0348ca005a7f60"}
Sep 11 18:28:15.299835 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.299712    2581 scope.go:117] "RemoveContainer" containerID="a82fa9e8051da366ddbfde93ffb7af0d42895541d054576c50f4c2927a597a70"
Sep 11 18:28:15.301149 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.301121    2581 generic.go:356] "Generic (PLEG): container finished" podID="6f695a9e-3528-41e7-a63d-969bebd396ac" containerID="5c029e3179a020be9b0c3d895e6ab590e4827eb21707d003d129b808c46a535e" exitCode=0
Sep 11 18:28:15.301262 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.301163    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"
Sep 11 18:28:15.301262 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.301225    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc" event={"ID":"6f695a9e-3528-41e7-a63d-969bebd396ac","Type":"ContainerDied","Data":"5c029e3179a020be9b0c3d895e6ab590e4827eb21707d003d129b808c46a535e"}
Sep 11 18:28:15.301351 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.301266    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc" event={"ID":"6f695a9e-3528-41e7-a63d-969bebd396ac","Type":"ContainerDied","Data":"26425d355aa2a1cd7f5f3687d1d220d77a376a16980a3670ca1b35e85a4a155f"}
Sep 11 18:28:15.311441 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.311303    2581 scope.go:117] "RemoveContainer" containerID="a82fa9e8051da366ddbfde93ffb7af0d42895541d054576c50f4c2927a597a70"
Sep 11 18:28:15.311722 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:28:15.311701    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82fa9e8051da366ddbfde93ffb7af0d42895541d054576c50f4c2927a597a70\": container with ID starting with a82fa9e8051da366ddbfde93ffb7af0d42895541d054576c50f4c2927a597a70 not found: ID does not exist" containerID="a82fa9e8051da366ddbfde93ffb7af0d42895541d054576c50f4c2927a597a70"
Sep 11 18:28:15.311810 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.311730    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82fa9e8051da366ddbfde93ffb7af0d42895541d054576c50f4c2927a597a70"} err="failed to get container status \"a82fa9e8051da366ddbfde93ffb7af0d42895541d054576c50f4c2927a597a70\": rpc error: code = NotFound desc = could not find container \"a82fa9e8051da366ddbfde93ffb7af0d42895541d054576c50f4c2927a597a70\": container with ID starting with a82fa9e8051da366ddbfde93ffb7af0d42895541d054576c50f4c2927a597a70 not found: ID does not exist"
Sep 11 18:28:15.311810 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.311750    2581 scope.go:117] "RemoveContainer" containerID="5c029e3179a020be9b0c3d895e6ab590e4827eb21707d003d129b808c46a535e"
Sep 11 18:28:15.320047 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.320033    2581 scope.go:117] "RemoveContainer" containerID="5c029e3179a020be9b0c3d895e6ab590e4827eb21707d003d129b808c46a535e"
Sep 11 18:28:15.320288 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:28:15.320275    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c029e3179a020be9b0c3d895e6ab590e4827eb21707d003d129b808c46a535e\": container with ID starting with 5c029e3179a020be9b0c3d895e6ab590e4827eb21707d003d129b808c46a535e not found: ID does not exist" containerID="5c029e3179a020be9b0c3d895e6ab590e4827eb21707d003d129b808c46a535e"
Sep 11 18:28:15.320330 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.320294    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c029e3179a020be9b0c3d895e6ab590e4827eb21707d003d129b808c46a535e"} err="failed to get container status \"5c029e3179a020be9b0c3d895e6ab590e4827eb21707d003d129b808c46a535e\": rpc error: code = NotFound desc = could not find container \"5c029e3179a020be9b0c3d895e6ab590e4827eb21707d003d129b808c46a535e\": container with ID starting with 5c029e3179a020be9b0c3d895e6ab590e4827eb21707d003d129b808c46a535e not found: ID does not exist"
Sep 11 18:28:15.330247 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.330227    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"]
Sep 11 18:28:15.337213 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.337192    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8370956b-66fd65cdd6-bn98k"]
Sep 11 18:28:15.349815 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.349796    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"]
Sep 11 18:28:15.355768 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.355741    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8370956b-5485554ff5-v4tvc"]
Sep 11 18:28:15.764208 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.764180    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25769e2e-02ea-4c99-bdc9-7278c6b5fc67" path="/var/lib/kubelet/pods/25769e2e-02ea-4c99-bdc9-7278c6b5fc67/volumes"
Sep 11 18:28:15.764614 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:15.764560    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f695a9e-3528-41e7-a63d-969bebd396ac" path="/var/lib/kubelet/pods/6f695a9e-3528-41e7-a63d-969bebd396ac/volumes"
Sep 11 18:28:21.139650 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.139620    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"]
Sep 11 18:28:21.140101 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.140088    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25769e2e-02ea-4c99-bdc9-7278c6b5fc67" containerName="payload-pre-processing"
Sep 11 18:28:21.140157 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.140103    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="25769e2e-02ea-4c99-bdc9-7278c6b5fc67" containerName="payload-pre-processing"
Sep 11 18:28:21.140157 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.140131    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f695a9e-3528-41e7-a63d-969bebd396ac" containerName="payload-processing"
Sep 11 18:28:21.140157 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.140142    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f695a9e-3528-41e7-a63d-969bebd396ac" containerName="payload-processing"
Sep 11 18:28:21.140288 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.140208    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f695a9e-3528-41e7-a63d-969bebd396ac" containerName="payload-processing"
Sep 11 18:28:21.140288 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.140220    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="25769e2e-02ea-4c99-bdc9-7278c6b5fc67" containerName="payload-pre-processing"
Sep 11 18:28:21.142364 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.142347    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"
Sep 11 18:28:21.144401 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.144358    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-839ba106\""
Sep 11 18:28:21.144638 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.144622    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-839ba106-dockercfg-hc8vf\""
Sep 11 18:28:21.151601 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.151577    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"]
Sep 11 18:28:21.161411 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.161386    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"]
Sep 11 18:28:21.164276 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.164259    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"
Sep 11 18:28:21.174093 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.174073    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"]
Sep 11 18:28:21.203919 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.203896    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6bcbb6d-18a4-4980-bc2e-f05e927c412f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x\" (UID: \"f6bcbb6d-18a4-4980-bc2e-f05e927c412f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"
Sep 11 18:28:21.204023 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.203939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pghp2\" (UniqueName: \"kubernetes.io/projected/f6bcbb6d-18a4-4980-bc2e-f05e927c412f-kube-api-access-pghp2\") pod \"payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x\" (UID: \"f6bcbb6d-18a4-4980-bc2e-f05e927c412f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"
Sep 11 18:28:21.304791 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.304761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pghp2\" (UniqueName: \"kubernetes.io/projected/f6bcbb6d-18a4-4980-bc2e-f05e927c412f-kube-api-access-pghp2\") pod \"payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x\" (UID: \"f6bcbb6d-18a4-4980-bc2e-f05e927c412f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"
Sep 11 18:28:21.304924 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.304832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e36dd0c-8ab8-47b8-9160-f9ecd707e38e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt\" (UID: \"6e36dd0c-8ab8-47b8-9160-f9ecd707e38e\") " pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"
Sep 11 18:28:21.304924 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.304872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvnxl\" (UniqueName: \"kubernetes.io/projected/6e36dd0c-8ab8-47b8-9160-f9ecd707e38e-kube-api-access-jvnxl\") pod \"payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt\" (UID: \"6e36dd0c-8ab8-47b8-9160-f9ecd707e38e\") " pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"
Sep 11 18:28:21.305040 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.304972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6bcbb6d-18a4-4980-bc2e-f05e927c412f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x\" (UID: \"f6bcbb6d-18a4-4980-bc2e-f05e927c412f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"
Sep 11 18:28:21.305636 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.305614    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6bcbb6d-18a4-4980-bc2e-f05e927c412f-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x\" (UID: \"f6bcbb6d-18a4-4980-bc2e-f05e927c412f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"
Sep 11 18:28:21.313651 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.313628    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pghp2\" (UniqueName: \"kubernetes.io/projected/f6bcbb6d-18a4-4980-bc2e-f05e927c412f-kube-api-access-pghp2\") pod \"payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x\" (UID: \"f6bcbb6d-18a4-4980-bc2e-f05e927c412f\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"
Sep 11 18:28:21.405447 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.405390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e36dd0c-8ab8-47b8-9160-f9ecd707e38e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt\" (UID: \"6e36dd0c-8ab8-47b8-9160-f9ecd707e38e\") " pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"
Sep 11 18:28:21.405447 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.405431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvnxl\" (UniqueName: \"kubernetes.io/projected/6e36dd0c-8ab8-47b8-9160-f9ecd707e38e-kube-api-access-jvnxl\") pod \"payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt\" (UID: \"6e36dd0c-8ab8-47b8-9160-f9ecd707e38e\") " pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"
Sep 11 18:28:21.406094 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.406077    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e36dd0c-8ab8-47b8-9160-f9ecd707e38e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt\" (UID: \"6e36dd0c-8ab8-47b8-9160-f9ecd707e38e\") " pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"
Sep 11 18:28:21.414666 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.414641    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvnxl\" (UniqueName: \"kubernetes.io/projected/6e36dd0c-8ab8-47b8-9160-f9ecd707e38e-kube-api-access-jvnxl\") pod \"payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt\" (UID: \"6e36dd0c-8ab8-47b8-9160-f9ecd707e38e\") " pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"
Sep 11 18:28:21.453558 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.453538    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"
Sep 11 18:28:21.476543 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.476522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"
Sep 11 18:28:21.725669 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.725646    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"]
Sep 11 18:28:21.726889 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:28:21.726862    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6bcbb6d_18a4_4980_bc2e_f05e927c412f.slice/crio-f26265e05d2581d8fe9581a95cd6e59202ce038d67dd4ed4bd9ee66b69a907ad WatchSource:0}: Error finding container f26265e05d2581d8fe9581a95cd6e59202ce038d67dd4ed4bd9ee66b69a907ad: Status 404 returned error can't find the container with id f26265e05d2581d8fe9581a95cd6e59202ce038d67dd4ed4bd9ee66b69a907ad
Sep 11 18:28:21.762784 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:28:21.762749    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e36dd0c_8ab8_47b8_9160_f9ecd707e38e.slice/crio-266ef6dc0541636201564eb9c253c04c19690cdabe4e50e8183e24fbdeeee213 WatchSource:0}: Error finding container 266ef6dc0541636201564eb9c253c04c19690cdabe4e50e8183e24fbdeeee213: Status 404 returned error can't find the container with id 266ef6dc0541636201564eb9c253c04c19690cdabe4e50e8183e24fbdeeee213
Sep 11 18:28:21.766217 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.766197    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"]
Sep 11 18:28:21.942200 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:21.942160    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:28:22.342765 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:22.342729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt" event={"ID":"6e36dd0c-8ab8-47b8-9160-f9ecd707e38e","Type":"ContainerStarted","Data":"b6b7e875a3494a78c83db034a2804aedc3a2d82523e9ff125297224e88b021f5"}
Sep 11 18:28:22.343202 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:22.342790    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt" event={"ID":"6e36dd0c-8ab8-47b8-9160-f9ecd707e38e","Type":"ContainerStarted","Data":"266ef6dc0541636201564eb9c253c04c19690cdabe4e50e8183e24fbdeeee213"}
Sep 11 18:28:22.343202 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:22.342850    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"
Sep 11 18:28:22.344592 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:22.344564    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x" event={"ID":"f6bcbb6d-18a4-4980-bc2e-f05e927c412f","Type":"ContainerStarted","Data":"765a21a84d2d2c52f86c67283021307318b75c716d624a5ac0d6b564d05124d1"}
Sep 11 18:28:22.344733 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:22.344603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x" event={"ID":"f6bcbb6d-18a4-4980-bc2e-f05e927c412f","Type":"ContainerStarted","Data":"f26265e05d2581d8fe9581a95cd6e59202ce038d67dd4ed4bd9ee66b69a907ad"}
Sep 11 18:28:22.344733 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:22.344659    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"
Sep 11 18:28:22.360403 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:22.360360    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt" podStartSLOduration=1.360347679 podStartE2EDuration="1.360347679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:22.358536712 +0000 UTC m=+779.099663556" watchObservedRunningTime="2026-09-11 18:28:22.360347679 +0000 UTC m=+779.101474519"
Sep 11 18:28:22.375625 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:22.375582    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x" podStartSLOduration=1.375569023 podStartE2EDuration="1.375569023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:22.37328575 +0000 UTC m=+779.114412594" watchObservedRunningTime="2026-09-11 18:28:22.375569023 +0000 UTC m=+779.116695865"
Sep 11 18:28:33.352931 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:33.352896    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"
Sep 11 18:28:33.353290 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:33.352966    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"
Sep 11 18:28:38.551134 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:38.551101    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:28:49.659052 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:49.658959    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"]
Sep 11 18:28:49.659652 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:49.659192    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt" podUID="6e36dd0c-8ab8-47b8-9160-f9ecd707e38e" containerName="payload-processing" containerID="cri-o://b6b7e875a3494a78c83db034a2804aedc3a2d82523e9ff125297224e88b021f5" gracePeriod=30
Sep 11 18:28:49.670943 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:49.670915    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"]
Sep 11 18:28:49.671128 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:49.671108    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x" podUID="f6bcbb6d-18a4-4980-bc2e-f05e927c412f" containerName="payload-pre-processing" containerID="cri-o://765a21a84d2d2c52f86c67283021307318b75c716d624a5ac0d6b564d05124d1" gracePeriod=30
Sep 11 18:28:49.932470 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:49.932445    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"
Sep 11 18:28:49.947705 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:49.947685    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"
Sep 11 18:28:49.962456 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:49.962432    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e36dd0c-8ab8-47b8-9160-f9ecd707e38e-plugins-config-volume\") pod \"6e36dd0c-8ab8-47b8-9160-f9ecd707e38e\" (UID: \"6e36dd0c-8ab8-47b8-9160-f9ecd707e38e\") "
Sep 11 18:28:49.962611 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:49.962464    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvnxl\" (UniqueName: \"kubernetes.io/projected/6e36dd0c-8ab8-47b8-9160-f9ecd707e38e-kube-api-access-jvnxl\") pod \"6e36dd0c-8ab8-47b8-9160-f9ecd707e38e\" (UID: \"6e36dd0c-8ab8-47b8-9160-f9ecd707e38e\") "
Sep 11 18:28:49.962830 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:49.962806    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e36dd0c-8ab8-47b8-9160-f9ecd707e38e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e36dd0c-8ab8-47b8-9160-f9ecd707e38e" (UID: "6e36dd0c-8ab8-47b8-9160-f9ecd707e38e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:28:49.965926 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:49.965903    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e36dd0c-8ab8-47b8-9160-f9ecd707e38e-kube-api-access-jvnxl" (OuterVolumeSpecName: "kube-api-access-jvnxl") pod "6e36dd0c-8ab8-47b8-9160-f9ecd707e38e" (UID: "6e36dd0c-8ab8-47b8-9160-f9ecd707e38e"). InnerVolumeSpecName "kube-api-access-jvnxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:28:50.063054 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.063023    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6bcbb6d-18a4-4980-bc2e-f05e927c412f-plugins-config-volume\") pod \"f6bcbb6d-18a4-4980-bc2e-f05e927c412f\" (UID: \"f6bcbb6d-18a4-4980-bc2e-f05e927c412f\") "
Sep 11 18:28:50.063219 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.063126    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pghp2\" (UniqueName: \"kubernetes.io/projected/f6bcbb6d-18a4-4980-bc2e-f05e927c412f-kube-api-access-pghp2\") pod \"f6bcbb6d-18a4-4980-bc2e-f05e927c412f\" (UID: \"f6bcbb6d-18a4-4980-bc2e-f05e927c412f\") "
Sep 11 18:28:50.063458 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.063431    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e36dd0c-8ab8-47b8-9160-f9ecd707e38e-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:28:50.063562 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.063465    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvnxl\" (UniqueName: \"kubernetes.io/projected/6e36dd0c-8ab8-47b8-9160-f9ecd707e38e-kube-api-access-jvnxl\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:28:50.063562 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.063432    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6bcbb6d-18a4-4980-bc2e-f05e927c412f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f6bcbb6d-18a4-4980-bc2e-f05e927c412f" (UID: "f6bcbb6d-18a4-4980-bc2e-f05e927c412f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:28:50.065308 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.065285    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6bcbb6d-18a4-4980-bc2e-f05e927c412f-kube-api-access-pghp2" (OuterVolumeSpecName: "kube-api-access-pghp2") pod "f6bcbb6d-18a4-4980-bc2e-f05e927c412f" (UID: "f6bcbb6d-18a4-4980-bc2e-f05e927c412f"). InnerVolumeSpecName "kube-api-access-pghp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:28:50.164056 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.164023    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pghp2\" (UniqueName: \"kubernetes.io/projected/f6bcbb6d-18a4-4980-bc2e-f05e927c412f-kube-api-access-pghp2\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:28:50.164056 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.164049    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6bcbb6d-18a4-4980-bc2e-f05e927c412f-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:28:50.491514 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.491462    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6bcbb6d-18a4-4980-bc2e-f05e927c412f" containerID="765a21a84d2d2c52f86c67283021307318b75c716d624a5ac0d6b564d05124d1" exitCode=0
Sep 11 18:28:50.491668 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.491515    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x" event={"ID":"f6bcbb6d-18a4-4980-bc2e-f05e927c412f","Type":"ContainerDied","Data":"765a21a84d2d2c52f86c67283021307318b75c716d624a5ac0d6b564d05124d1"}
Sep 11 18:28:50.491668 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.491562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x" event={"ID":"f6bcbb6d-18a4-4980-bc2e-f05e927c412f","Type":"ContainerDied","Data":"f26265e05d2581d8fe9581a95cd6e59202ce038d67dd4ed4bd9ee66b69a907ad"}
Sep 11 18:28:50.491668 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.491583    2581 scope.go:117] "RemoveContainer" containerID="765a21a84d2d2c52f86c67283021307318b75c716d624a5ac0d6b564d05124d1"
Sep 11 18:28:50.491668 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.491526    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"
Sep 11 18:28:50.493204 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.493184    2581 generic.go:356] "Generic (PLEG): container finished" podID="6e36dd0c-8ab8-47b8-9160-f9ecd707e38e" containerID="b6b7e875a3494a78c83db034a2804aedc3a2d82523e9ff125297224e88b021f5" exitCode=0
Sep 11 18:28:50.493341 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.493216    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt" event={"ID":"6e36dd0c-8ab8-47b8-9160-f9ecd707e38e","Type":"ContainerDied","Data":"b6b7e875a3494a78c83db034a2804aedc3a2d82523e9ff125297224e88b021f5"}
Sep 11 18:28:50.493341 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.493224    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"
Sep 11 18:28:50.493341 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.493247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt" event={"ID":"6e36dd0c-8ab8-47b8-9160-f9ecd707e38e","Type":"ContainerDied","Data":"266ef6dc0541636201564eb9c253c04c19690cdabe4e50e8183e24fbdeeee213"}
Sep 11 18:28:50.501407 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.501394    2581 scope.go:117] "RemoveContainer" containerID="765a21a84d2d2c52f86c67283021307318b75c716d624a5ac0d6b564d05124d1"
Sep 11 18:28:50.501738 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:28:50.501705    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765a21a84d2d2c52f86c67283021307318b75c716d624a5ac0d6b564d05124d1\": container with ID starting with 765a21a84d2d2c52f86c67283021307318b75c716d624a5ac0d6b564d05124d1 not found: ID does not exist" containerID="765a21a84d2d2c52f86c67283021307318b75c716d624a5ac0d6b564d05124d1"
Sep 11 18:28:50.501847 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.501748    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765a21a84d2d2c52f86c67283021307318b75c716d624a5ac0d6b564d05124d1"} err="failed to get container status \"765a21a84d2d2c52f86c67283021307318b75c716d624a5ac0d6b564d05124d1\": rpc error: code = NotFound desc = could not find container \"765a21a84d2d2c52f86c67283021307318b75c716d624a5ac0d6b564d05124d1\": container with ID starting with 765a21a84d2d2c52f86c67283021307318b75c716d624a5ac0d6b564d05124d1 not found: ID does not exist"
Sep 11 18:28:50.501847 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.501771    2581 scope.go:117] "RemoveContainer" containerID="b6b7e875a3494a78c83db034a2804aedc3a2d82523e9ff125297224e88b021f5"
Sep 11 18:28:50.509765 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.509747    2581 scope.go:117] "RemoveContainer" containerID="b6b7e875a3494a78c83db034a2804aedc3a2d82523e9ff125297224e88b021f5"
Sep 11 18:28:50.510017 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:28:50.510001    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b7e875a3494a78c83db034a2804aedc3a2d82523e9ff125297224e88b021f5\": container with ID starting with b6b7e875a3494a78c83db034a2804aedc3a2d82523e9ff125297224e88b021f5 not found: ID does not exist" containerID="b6b7e875a3494a78c83db034a2804aedc3a2d82523e9ff125297224e88b021f5"
Sep 11 18:28:50.510079 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.510024    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b7e875a3494a78c83db034a2804aedc3a2d82523e9ff125297224e88b021f5"} err="failed to get container status \"b6b7e875a3494a78c83db034a2804aedc3a2d82523e9ff125297224e88b021f5\": rpc error: code = NotFound desc = could not find container \"b6b7e875a3494a78c83db034a2804aedc3a2d82523e9ff125297224e88b021f5\": container with ID starting with b6b7e875a3494a78c83db034a2804aedc3a2d82523e9ff125297224e88b021f5 not found: ID does not exist"
Sep 11 18:28:50.516300 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.516276    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"]
Sep 11 18:28:50.520830 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.520811    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-839ba106-5f985ddf94-h2vvt"]
Sep 11 18:28:50.531758 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.531739    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"]
Sep 11 18:28:50.538894 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:50.538831    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-839ba106-d6774765d-rq68x"]
Sep 11 18:28:51.764734 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:51.764700    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e36dd0c-8ab8-47b8-9160-f9ecd707e38e" path="/var/lib/kubelet/pods/6e36dd0c-8ab8-47b8-9160-f9ecd707e38e/volumes"
Sep 11 18:28:51.765076 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:51.765034    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6bcbb6d-18a4-4980-bc2e-f05e927c412f" path="/var/lib/kubelet/pods/f6bcbb6d-18a4-4980-bc2e-f05e927c412f/volumes"
Sep 11 18:28:53.930922 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:53.930891    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"]
Sep 11 18:28:53.931306 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:53.931292    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e36dd0c-8ab8-47b8-9160-f9ecd707e38e" containerName="payload-processing"
Sep 11 18:28:53.931349 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:53.931307    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e36dd0c-8ab8-47b8-9160-f9ecd707e38e" containerName="payload-processing"
Sep 11 18:28:53.931349 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:53.931330    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6bcbb6d-18a4-4980-bc2e-f05e927c412f" containerName="payload-pre-processing"
Sep 11 18:28:53.931349 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:53.931335    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6bcbb6d-18a4-4980-bc2e-f05e927c412f" containerName="payload-pre-processing"
Sep 11 18:28:53.931441 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:53.931400    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6bcbb6d-18a4-4980-bc2e-f05e927c412f" containerName="payload-pre-processing"
Sep 11 18:28:53.931441 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:53.931408    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e36dd0c-8ab8-47b8-9160-f9ecd707e38e" containerName="payload-processing"
Sep 11 18:28:53.934297 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:53.934282    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:53.937026 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:53.936994    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-efa8b9\"/\"default-dockercfg-gn54h\""
Sep 11 18:28:53.937026 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:53.937010    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-efa8b9\"/\"openshift-service-ca.crt\""
Sep 11 18:28:53.937222 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:53.937071    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-efa8b9\"/\"simulator-w0-efa8b9-kserve-self-signed-certs\""
Sep 11 18:28:53.937278 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:53.937254    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-efa8b9\"/\"kube-root-ca.crt\""
Sep 11 18:28:53.942234 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:53.942215    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"]
Sep 11 18:28:54.002368 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.002327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4drtl\" (UniqueName: \"kubernetes.io/projected/037f2be3-9585-409a-adbc-7ca76eb13efc-kube-api-access-4drtl\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.002554 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.002388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-model-cache\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.002554 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.002458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-home\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.002554 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.002522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-dshm\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.002691 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.002562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/037f2be3-9585-409a-adbc-7ca76eb13efc-tls-certs\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.002691 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.002581    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-tmp-dir\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.103773 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.103742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-dshm\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.103927 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.103788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/037f2be3-9585-409a-adbc-7ca76eb13efc-tls-certs\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.103927 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.103910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-tmp-dir\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.104026 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.104004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4drtl\" (UniqueName: \"kubernetes.io/projected/037f2be3-9585-409a-adbc-7ca76eb13efc-kube-api-access-4drtl\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.104081 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.104063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-model-cache\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.104133 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.104112    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-home\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.104312 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.104275    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-tmp-dir\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.104399 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.104381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-model-cache\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.104705 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.104685    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-home\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.106094 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.106071    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-dshm\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.106405 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.106389    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/037f2be3-9585-409a-adbc-7ca76eb13efc-tls-certs\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.115737 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.115719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4drtl\" (UniqueName: \"kubernetes.io/projected/037f2be3-9585-409a-adbc-7ca76eb13efc-kube-api-access-4drtl\") pod \"simulator-w0-efa8b9-kserve-5d5577f696-zc4g6\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") " pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.244561 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.244465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:28:54.515443 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:54.515418    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"]
Sep 11 18:28:54.516390 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:28:54.516367    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod037f2be3_9585_409a_adbc_7ca76eb13efc.slice/crio-f5e74bd57516c121c6a97ae69eb22202903f9f43ebc54d59de2e53e15d4c0e1b WatchSource:0}: Error finding container f5e74bd57516c121c6a97ae69eb22202903f9f43ebc54d59de2e53e15d4c0e1b: Status 404 returned error can't find the container with id f5e74bd57516c121c6a97ae69eb22202903f9f43ebc54d59de2e53e15d4c0e1b
Sep 11 18:28:55.525680 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:55.525638    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6" event={"ID":"037f2be3-9585-409a-adbc-7ca76eb13efc","Type":"ContainerStarted","Data":"8141f2b74570ef5533246946238064db73a314d1811b87c9a1cecdf770065105"}
Sep 11 18:28:55.526133 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:55.525691    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6" event={"ID":"037f2be3-9585-409a-adbc-7ca76eb13efc","Type":"ContainerStarted","Data":"f5e74bd57516c121c6a97ae69eb22202903f9f43ebc54d59de2e53e15d4c0e1b"}
Sep 11 18:28:55.546241 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:55.546196    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6" podStartSLOduration=2.546179559 podStartE2EDuration="2.546179559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:55.54450371 +0000 UTC m=+812.285630544" watchObservedRunningTime="2026-09-11 18:28:55.546179559 +0000 UTC m=+812.287306413"
Sep 11 18:28:55.740739 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:55.740709    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:28:56.262667 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.262631    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"]
Sep 11 18:28:56.265402 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.265374    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"
Sep 11 18:28:56.269235 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.269210    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3565479f\""
Sep 11 18:28:56.269409 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.269389    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3565479f-dockercfg-g9xj5\""
Sep 11 18:28:56.277852 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.277831    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"]
Sep 11 18:28:56.280647 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.280633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"
Sep 11 18:28:56.285694 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.285673    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"]
Sep 11 18:28:56.298173 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.298110    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"]
Sep 11 18:28:56.326164 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.326138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4rnp\" (UniqueName: \"kubernetes.io/projected/3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c-kube-api-access-g4rnp\") pod \"payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5\" (UID: \"3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"
Sep 11 18:28:56.326249 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.326236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh89x\" (UniqueName: \"kubernetes.io/projected/73be478e-7b7c-4fd3-a5d4-fac5d02a5cce-kube-api-access-nh89x\") pod \"payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns\" (UID: \"73be478e-7b7c-4fd3-a5d4-fac5d02a5cce\") " pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"
Sep 11 18:28:56.326296 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.326254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5\" (UID: \"3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"
Sep 11 18:28:56.326296 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.326280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73be478e-7b7c-4fd3-a5d4-fac5d02a5cce-plugins-config-volume\") pod \"payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns\" (UID: \"73be478e-7b7c-4fd3-a5d4-fac5d02a5cce\") " pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"
Sep 11 18:28:56.427267 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.427235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73be478e-7b7c-4fd3-a5d4-fac5d02a5cce-plugins-config-volume\") pod \"payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns\" (UID: \"73be478e-7b7c-4fd3-a5d4-fac5d02a5cce\") " pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"
Sep 11 18:28:56.427396 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.427276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4rnp\" (UniqueName: \"kubernetes.io/projected/3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c-kube-api-access-g4rnp\") pod \"payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5\" (UID: \"3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"
Sep 11 18:28:56.427396 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.427353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh89x\" (UniqueName: \"kubernetes.io/projected/73be478e-7b7c-4fd3-a5d4-fac5d02a5cce-kube-api-access-nh89x\") pod \"payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns\" (UID: \"73be478e-7b7c-4fd3-a5d4-fac5d02a5cce\") " pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"
Sep 11 18:28:56.427396 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.427375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5\" (UID: \"3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"
Sep 11 18:28:56.428021 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.428002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73be478e-7b7c-4fd3-a5d4-fac5d02a5cce-plugins-config-volume\") pod \"payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns\" (UID: \"73be478e-7b7c-4fd3-a5d4-fac5d02a5cce\") " pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"
Sep 11 18:28:56.428076 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.428059    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5\" (UID: \"3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"
Sep 11 18:28:56.435752 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.435726    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh89x\" (UniqueName: \"kubernetes.io/projected/73be478e-7b7c-4fd3-a5d4-fac5d02a5cce-kube-api-access-nh89x\") pod \"payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns\" (UID: \"73be478e-7b7c-4fd3-a5d4-fac5d02a5cce\") " pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"
Sep 11 18:28:56.435855 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.435820    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4rnp\" (UniqueName: \"kubernetes.io/projected/3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c-kube-api-access-g4rnp\") pod \"payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5\" (UID: \"3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"
Sep 11 18:28:56.577497 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.577401    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"
Sep 11 18:28:56.594239 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.594215    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"
Sep 11 18:28:56.854549 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.854514    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"]
Sep 11 18:28:56.856104 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:28:56.856068    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5db3fb_9e08_4ba5_bef7_8a7eaab0f72c.slice/crio-066f87a50c99d642021413e04b638e533507a0ec06616b66904740bb2d00e857 WatchSource:0}: Error finding container 066f87a50c99d642021413e04b638e533507a0ec06616b66904740bb2d00e857: Status 404 returned error can't find the container with id 066f87a50c99d642021413e04b638e533507a0ec06616b66904740bb2d00e857
Sep 11 18:28:56.870987 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:56.870963    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"]
Sep 11 18:28:57.539112 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:57.539079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns" event={"ID":"73be478e-7b7c-4fd3-a5d4-fac5d02a5cce","Type":"ContainerStarted","Data":"552a1b668d069b6310428c8f392e60fc046c12f1a7e5a5448dff7b563c6a2959"}
Sep 11 18:28:57.539317 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:57.539138    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"
Sep 11 18:28:57.539317 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:57.539159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns" event={"ID":"73be478e-7b7c-4fd3-a5d4-fac5d02a5cce","Type":"ContainerStarted","Data":"f03d7f2f9fb59693e3303d5bee187589c2398d88ab4f7528e194f4a808996f22"}
Sep 11 18:28:57.540694 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:57.540667    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5" event={"ID":"3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c","Type":"ContainerStarted","Data":"5da5c1d8e1ef8948f99b291ef9cfb33b182eeffe9f029e6358ed81677dff5283"}
Sep 11 18:28:57.540826 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:57.540709    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5" event={"ID":"3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c","Type":"ContainerStarted","Data":"066f87a50c99d642021413e04b638e533507a0ec06616b66904740bb2d00e857"}
Sep 11 18:28:57.540826 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:57.540751    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"
Sep 11 18:28:57.558618 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:57.558574    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns" podStartSLOduration=1.558560211 podStartE2EDuration="1.558560211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:57.556285012 +0000 UTC m=+814.297411867" watchObservedRunningTime="2026-09-11 18:28:57.558560211 +0000 UTC m=+814.299687110"
Sep 11 18:28:57.577003 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:28:57.576961    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5" podStartSLOduration=1.576948229 podStartE2EDuration="1.576948229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:28:57.57552596 +0000 UTC m=+814.316652815" watchObservedRunningTime="2026-09-11 18:28:57.576948229 +0000 UTC m=+814.318075072"
Sep 11 18:29:04.245208 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:04.245174    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:29:04.245661 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:04.245218    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:29:04.257753 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:04.257731    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:29:04.588109 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:04.588028    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:29:08.548584 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:08.548549    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"
Sep 11 18:29:08.548962 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:08.548677    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"
Sep 11 18:29:13.442332 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:13.442279    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:29:25.302465 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.302431    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"]
Sep 11 18:29:25.302962 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.302753    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns" podUID="73be478e-7b7c-4fd3-a5d4-fac5d02a5cce" containerName="payload-processing" containerID="cri-o://552a1b668d069b6310428c8f392e60fc046c12f1a7e5a5448dff7b563c6a2959" gracePeriod=30
Sep 11 18:29:25.308929 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.308904    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"]
Sep 11 18:29:25.309133 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.309111    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5" podUID="3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c" containerName="payload-pre-processing" containerID="cri-o://5da5c1d8e1ef8948f99b291ef9cfb33b182eeffe9f029e6358ed81677dff5283" gracePeriod=30
Sep 11 18:29:25.588962 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.588934    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"
Sep 11 18:29:25.591568 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.591548    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nh89x\" (UniqueName: \"kubernetes.io/projected/73be478e-7b7c-4fd3-a5d4-fac5d02a5cce-kube-api-access-nh89x\") pod \"73be478e-7b7c-4fd3-a5d4-fac5d02a5cce\" (UID: \"73be478e-7b7c-4fd3-a5d4-fac5d02a5cce\") "
Sep 11 18:29:25.591670 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.591602    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73be478e-7b7c-4fd3-a5d4-fac5d02a5cce-plugins-config-volume\") pod \"73be478e-7b7c-4fd3-a5d4-fac5d02a5cce\" (UID: \"73be478e-7b7c-4fd3-a5d4-fac5d02a5cce\") "
Sep 11 18:29:25.591954 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.591933    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"
Sep 11 18:29:25.592050 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.591958    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73be478e-7b7c-4fd3-a5d4-fac5d02a5cce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73be478e-7b7c-4fd3-a5d4-fac5d02a5cce" (UID: "73be478e-7b7c-4fd3-a5d4-fac5d02a5cce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:29:25.593689 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.593671    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73be478e-7b7c-4fd3-a5d4-fac5d02a5cce-kube-api-access-nh89x" (OuterVolumeSpecName: "kube-api-access-nh89x") pod "73be478e-7b7c-4fd3-a5d4-fac5d02a5cce" (UID: "73be478e-7b7c-4fd3-a5d4-fac5d02a5cce"). InnerVolumeSpecName "kube-api-access-nh89x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:25.688740 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.688710    2581 generic.go:356] "Generic (PLEG): container finished" podID="3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c" containerID="5da5c1d8e1ef8948f99b291ef9cfb33b182eeffe9f029e6358ed81677dff5283" exitCode=0
Sep 11 18:29:25.688902 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.688755    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"
Sep 11 18:29:25.688902 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.688803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5" event={"ID":"3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c","Type":"ContainerDied","Data":"5da5c1d8e1ef8948f99b291ef9cfb33b182eeffe9f029e6358ed81677dff5283"}
Sep 11 18:29:25.688902 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.688852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5" event={"ID":"3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c","Type":"ContainerDied","Data":"066f87a50c99d642021413e04b638e533507a0ec06616b66904740bb2d00e857"}
Sep 11 18:29:25.688902 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.688879    2581 scope.go:117] "RemoveContainer" containerID="5da5c1d8e1ef8948f99b291ef9cfb33b182eeffe9f029e6358ed81677dff5283"
Sep 11 18:29:25.690096 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.690077    2581 generic.go:356] "Generic (PLEG): container finished" podID="73be478e-7b7c-4fd3-a5d4-fac5d02a5cce" containerID="552a1b668d069b6310428c8f392e60fc046c12f1a7e5a5448dff7b563c6a2959" exitCode=0
Sep 11 18:29:25.690188 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.690122    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"
Sep 11 18:29:25.690188 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.690132    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns" event={"ID":"73be478e-7b7c-4fd3-a5d4-fac5d02a5cce","Type":"ContainerDied","Data":"552a1b668d069b6310428c8f392e60fc046c12f1a7e5a5448dff7b563c6a2959"}
Sep 11 18:29:25.690188 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.690179    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns" event={"ID":"73be478e-7b7c-4fd3-a5d4-fac5d02a5cce","Type":"ContainerDied","Data":"f03d7f2f9fb59693e3303d5bee187589c2398d88ab4f7528e194f4a808996f22"}
Sep 11 18:29:25.692414 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.692391    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c-plugins-config-volume\") pod \"3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c\" (UID: \"3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c\") "
Sep 11 18:29:25.692563 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.692441    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4rnp\" (UniqueName: \"kubernetes.io/projected/3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c-kube-api-access-g4rnp\") pod \"3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c\" (UID: \"3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c\") "
Sep 11 18:29:25.692818 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.692794    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh89x\" (UniqueName: \"kubernetes.io/projected/73be478e-7b7c-4fd3-a5d4-fac5d02a5cce-kube-api-access-nh89x\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:29:25.692918 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.692825    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73be478e-7b7c-4fd3-a5d4-fac5d02a5cce-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:29:25.692918 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.692793    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c" (UID: "3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:29:25.694751 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.694727    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c-kube-api-access-g4rnp" (OuterVolumeSpecName: "kube-api-access-g4rnp") pod "3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c" (UID: "3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c"). InnerVolumeSpecName "kube-api-access-g4rnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:29:25.698538 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.698520    2581 scope.go:117] "RemoveContainer" containerID="5da5c1d8e1ef8948f99b291ef9cfb33b182eeffe9f029e6358ed81677dff5283"
Sep 11 18:29:25.698799 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:29:25.698783    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da5c1d8e1ef8948f99b291ef9cfb33b182eeffe9f029e6358ed81677dff5283\": container with ID starting with 5da5c1d8e1ef8948f99b291ef9cfb33b182eeffe9f029e6358ed81677dff5283 not found: ID does not exist" containerID="5da5c1d8e1ef8948f99b291ef9cfb33b182eeffe9f029e6358ed81677dff5283"
Sep 11 18:29:25.698849 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.698807    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da5c1d8e1ef8948f99b291ef9cfb33b182eeffe9f029e6358ed81677dff5283"} err="failed to get container status \"5da5c1d8e1ef8948f99b291ef9cfb33b182eeffe9f029e6358ed81677dff5283\": rpc error: code = NotFound desc = could not find container \"5da5c1d8e1ef8948f99b291ef9cfb33b182eeffe9f029e6358ed81677dff5283\": container with ID starting with 5da5c1d8e1ef8948f99b291ef9cfb33b182eeffe9f029e6358ed81677dff5283 not found: ID does not exist"
Sep 11 18:29:25.698849 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.698824    2581 scope.go:117] "RemoveContainer" containerID="552a1b668d069b6310428c8f392e60fc046c12f1a7e5a5448dff7b563c6a2959"
Sep 11 18:29:25.706656 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.706627    2581 scope.go:117] "RemoveContainer" containerID="552a1b668d069b6310428c8f392e60fc046c12f1a7e5a5448dff7b563c6a2959"
Sep 11 18:29:25.706893 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:29:25.706875    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552a1b668d069b6310428c8f392e60fc046c12f1a7e5a5448dff7b563c6a2959\": container with ID starting with 552a1b668d069b6310428c8f392e60fc046c12f1a7e5a5448dff7b563c6a2959 not found: ID does not exist" containerID="552a1b668d069b6310428c8f392e60fc046c12f1a7e5a5448dff7b563c6a2959"
Sep 11 18:29:25.706941 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.706898    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552a1b668d069b6310428c8f392e60fc046c12f1a7e5a5448dff7b563c6a2959"} err="failed to get container status \"552a1b668d069b6310428c8f392e60fc046c12f1a7e5a5448dff7b563c6a2959\": rpc error: code = NotFound desc = could not find container \"552a1b668d069b6310428c8f392e60fc046c12f1a7e5a5448dff7b563c6a2959\": container with ID starting with 552a1b668d069b6310428c8f392e60fc046c12f1a7e5a5448dff7b563c6a2959 not found: ID does not exist"
Sep 11 18:29:25.711764 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.711743    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"]
Sep 11 18:29:25.715888 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.715868    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3565479f-fc4866c8f-mzpns"]
Sep 11 18:29:25.764439 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.764412    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73be478e-7b7c-4fd3-a5d4-fac5d02a5cce" path="/var/lib/kubelet/pods/73be478e-7b7c-4fd3-a5d4-fac5d02a5cce/volumes"
Sep 11 18:29:25.793257 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.793238    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:29:25.793355 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:25.793257    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4rnp\" (UniqueName: \"kubernetes.io/projected/3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c-kube-api-access-g4rnp\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:29:25.805883 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:29:25.805858    2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5db3fb_9e08_4ba5_bef7_8a7eaab0f72c.slice/crio-066f87a50c99d642021413e04b638e533507a0ec06616b66904740bb2d00e857\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5db3fb_9e08_4ba5_bef7_8a7eaab0f72c.slice\": RecentStats: unable to find data in memory cache]"
Sep 11 18:29:26.005527 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:26.005494    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"]
Sep 11 18:29:26.012462 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:26.012431    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3565479f-5c88f9cb86-tzhj5"]
Sep 11 18:29:27.678225 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:27.678192    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:29:27.765031 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:27.764999    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c" path="/var/lib/kubelet/pods/3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c/volumes"
Sep 11 18:29:32.167310 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.167273    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"]
Sep 11 18:29:32.167906 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.167885    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73be478e-7b7c-4fd3-a5d4-fac5d02a5cce" containerName="payload-processing"
Sep 11 18:29:32.167993 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.167909    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="73be478e-7b7c-4fd3-a5d4-fac5d02a5cce" containerName="payload-processing"
Sep 11 18:29:32.167993 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.167931    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c" containerName="payload-pre-processing"
Sep 11 18:29:32.167993 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.167939    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c" containerName="payload-pre-processing"
Sep 11 18:29:32.168142 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.168068    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5db3fb-9e08-4ba5-bef7-8a7eaab0f72c" containerName="payload-pre-processing"
Sep 11 18:29:32.168142 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.168081    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="73be478e-7b7c-4fd3-a5d4-fac5d02a5cce" containerName="payload-processing"
Sep 11 18:29:32.172911 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.172891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"
Sep 11 18:29:32.175332 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.175313    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-68ac8212\""
Sep 11 18:29:32.180938 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.180917    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"]
Sep 11 18:29:32.192218 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.192196    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"]
Sep 11 18:29:32.196453 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.196438    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"
Sep 11 18:29:32.198678 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.198661    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-68ac8212-dockercfg-52f9v\""
Sep 11 18:29:32.205036 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.204708    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"]
Sep 11 18:29:32.255970 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.255937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n226d\" (UniqueName: \"kubernetes.io/projected/9e7605d7-416c-4ddd-a202-45004a2e446f-kube-api-access-n226d\") pod \"payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v\" (UID: \"9e7605d7-416c-4ddd-a202-45004a2e446f\") " pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"
Sep 11 18:29:32.256120 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.255973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce93669-4b71-4289-a4c6-52b1660d9d06-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd\" (UID: \"2ce93669-4b71-4289-a4c6-52b1660d9d06\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"
Sep 11 18:29:32.256120 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.256064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7605d7-416c-4ddd-a202-45004a2e446f-plugins-config-volume\") pod \"payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v\" (UID: \"9e7605d7-416c-4ddd-a202-45004a2e446f\") " pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"
Sep 11 18:29:32.256209 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.256150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnptq\" (UniqueName: \"kubernetes.io/projected/2ce93669-4b71-4289-a4c6-52b1660d9d06-kube-api-access-fnptq\") pod \"payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd\" (UID: \"2ce93669-4b71-4289-a4c6-52b1660d9d06\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"
Sep 11 18:29:32.357191 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.357161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnptq\" (UniqueName: \"kubernetes.io/projected/2ce93669-4b71-4289-a4c6-52b1660d9d06-kube-api-access-fnptq\") pod \"payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd\" (UID: \"2ce93669-4b71-4289-a4c6-52b1660d9d06\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"
Sep 11 18:29:32.357338 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.357220    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n226d\" (UniqueName: \"kubernetes.io/projected/9e7605d7-416c-4ddd-a202-45004a2e446f-kube-api-access-n226d\") pod \"payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v\" (UID: \"9e7605d7-416c-4ddd-a202-45004a2e446f\") " pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"
Sep 11 18:29:32.357338 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.357236    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce93669-4b71-4289-a4c6-52b1660d9d06-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd\" (UID: \"2ce93669-4b71-4289-a4c6-52b1660d9d06\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"
Sep 11 18:29:32.357428 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.357377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7605d7-416c-4ddd-a202-45004a2e446f-plugins-config-volume\") pod \"payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v\" (UID: \"9e7605d7-416c-4ddd-a202-45004a2e446f\") " pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"
Sep 11 18:29:32.357873 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.357856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce93669-4b71-4289-a4c6-52b1660d9d06-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd\" (UID: \"2ce93669-4b71-4289-a4c6-52b1660d9d06\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"
Sep 11 18:29:32.357982 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.357966    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7605d7-416c-4ddd-a202-45004a2e446f-plugins-config-volume\") pod \"payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v\" (UID: \"9e7605d7-416c-4ddd-a202-45004a2e446f\") " pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"
Sep 11 18:29:32.366548 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.366525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnptq\" (UniqueName: \"kubernetes.io/projected/2ce93669-4b71-4289-a4c6-52b1660d9d06-kube-api-access-fnptq\") pod \"payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd\" (UID: \"2ce93669-4b71-4289-a4c6-52b1660d9d06\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"
Sep 11 18:29:32.366658 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.366525    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n226d\" (UniqueName: \"kubernetes.io/projected/9e7605d7-416c-4ddd-a202-45004a2e446f-kube-api-access-n226d\") pod \"payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v\" (UID: \"9e7605d7-416c-4ddd-a202-45004a2e446f\") " pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"
Sep 11 18:29:32.483739 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.483670    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"
Sep 11 18:29:32.510695 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.510672    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"
Sep 11 18:29:32.767137 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.767114    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"]
Sep 11 18:29:32.768157 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:29:32.768135    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ce93669_4b71_4289_a4c6_52b1660d9d06.slice/crio-49b70ce731e0bcd724a0fbb7745f29d3a1c4a37b47804e68cb401fc3a9cd41f9 WatchSource:0}: Error finding container 49b70ce731e0bcd724a0fbb7745f29d3a1c4a37b47804e68cb401fc3a9cd41f9: Status 404 returned error can't find the container with id 49b70ce731e0bcd724a0fbb7745f29d3a1c4a37b47804e68cb401fc3a9cd41f9
Sep 11 18:29:32.818787 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:32.818760    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"]
Sep 11 18:29:32.822840 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:29:32.822819    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7605d7_416c_4ddd_a202_45004a2e446f.slice/crio-fbdaf63e691e0ba49e2ca9e5ed6ba3d8730bf192e240598f2bce16e4b1d07639 WatchSource:0}: Error finding container fbdaf63e691e0ba49e2ca9e5ed6ba3d8730bf192e240598f2bce16e4b1d07639: Status 404 returned error can't find the container with id fbdaf63e691e0ba49e2ca9e5ed6ba3d8730bf192e240598f2bce16e4b1d07639
Sep 11 18:29:33.734376 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:33.734332    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v" event={"ID":"9e7605d7-416c-4ddd-a202-45004a2e446f","Type":"ContainerStarted","Data":"15dd8a649e884db211f5662276a43906f7b4a9c20eb321ff76030cd9b5faf981"}
Sep 11 18:29:33.734860 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:33.734420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v" event={"ID":"9e7605d7-416c-4ddd-a202-45004a2e446f","Type":"ContainerStarted","Data":"fbdaf63e691e0ba49e2ca9e5ed6ba3d8730bf192e240598f2bce16e4b1d07639"}
Sep 11 18:29:33.734860 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:33.734444    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"
Sep 11 18:29:33.735921 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:33.735896    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd" event={"ID":"2ce93669-4b71-4289-a4c6-52b1660d9d06","Type":"ContainerStarted","Data":"cf91004cdef42735be82b145eef0ac98ed12b81b681c20131fbb6ba147239b95"}
Sep 11 18:29:33.736024 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:33.735939    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd" event={"ID":"2ce93669-4b71-4289-a4c6-52b1660d9d06","Type":"ContainerStarted","Data":"49b70ce731e0bcd724a0fbb7745f29d3a1c4a37b47804e68cb401fc3a9cd41f9"}
Sep 11 18:29:33.736024 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:33.735966    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"
Sep 11 18:29:33.765643 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:33.765600    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v" podStartSLOduration=1.765587478 podStartE2EDuration="1.765587478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:33.749844024 +0000 UTC m=+850.490970868" watchObservedRunningTime="2026-09-11 18:29:33.765587478 +0000 UTC m=+850.506714320"
Sep 11 18:29:33.767371 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:33.767340    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd" podStartSLOduration=1.767333139 podStartE2EDuration="1.767333139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:33.764529683 +0000 UTC m=+850.505656528" watchObservedRunningTime="2026-09-11 18:29:33.767333139 +0000 UTC m=+850.508459979"
Sep 11 18:29:36.625981 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.625949    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"]
Sep 11 18:29:36.629739 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.629723    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.632666 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.632639    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7e8ab9\"/\"default-dockercfg-ch6tb\""
Sep 11 18:29:36.632797 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.632641    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-7e8ab9\"/\"autores-model-7e8ab9-kserve-self-signed-certs\""
Sep 11 18:29:36.632797 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.632733    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7e8ab9\"/\"openshift-service-ca.crt\""
Sep 11 18:29:36.632929 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.632857    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-7e8ab9\"/\"kube-root-ca.crt\""
Sep 11 18:29:36.639170 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.639149    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"]
Sep 11 18:29:36.700295 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.700268    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26013c4-2ea1-4906-b4dc-4f27c30dac82-tls-certs\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.700415 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.700302    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp8t4\" (UniqueName: \"kubernetes.io/projected/e26013c4-2ea1-4906-b4dc-4f27c30dac82-kube-api-access-rp8t4\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.700415 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.700380    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-tmp-dir\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.700547 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.700432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-home\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.700547 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.700464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-dshm\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.700547 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.700531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-model-cache\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.801173 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.801143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-tmp-dir\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.801318 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.801188    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-home\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.801318 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.801228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-dshm\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.801318 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.801258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-model-cache\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.801517 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.801340    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26013c4-2ea1-4906-b4dc-4f27c30dac82-tls-certs\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.801620 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.801591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp8t4\" (UniqueName: \"kubernetes.io/projected/e26013c4-2ea1-4906-b4dc-4f27c30dac82-kube-api-access-rp8t4\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.801676 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.801639    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-home\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.801676 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.801593    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-tmp-dir\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.801818 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.801801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-model-cache\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.803923 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.803901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-dshm\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.804094 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.804074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26013c4-2ea1-4906-b4dc-4f27c30dac82-tls-certs\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.809796 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.809768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp8t4\" (UniqueName: \"kubernetes.io/projected/e26013c4-2ea1-4906-b4dc-4f27c30dac82-kube-api-access-rp8t4\") pod \"autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") " pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:36.941065 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:36.940977    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:37.223595 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:37.223571    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"]
Sep 11 18:29:37.225493 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:29:37.225452    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode26013c4_2ea1_4906_b4dc_4f27c30dac82.slice/crio-6456d156cee0faf7e14d8f5b755465c0368195c182ebc173c45172e43940260b WatchSource:0}: Error finding container 6456d156cee0faf7e14d8f5b755465c0368195c182ebc173c45172e43940260b: Status 404 returned error can't find the container with id 6456d156cee0faf7e14d8f5b755465c0368195c182ebc173c45172e43940260b
Sep 11 18:29:37.768188 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:37.768155    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt" event={"ID":"e26013c4-2ea1-4906-b4dc-4f27c30dac82","Type":"ContainerStarted","Data":"e86026c1839b4b07cb58a3ee572c3ee1f83279f90ef9604065ee942e440c6a97"}
Sep 11 18:29:37.768188 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:37.768193    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt" event={"ID":"e26013c4-2ea1-4906-b4dc-4f27c30dac82","Type":"ContainerStarted","Data":"6456d156cee0faf7e14d8f5b755465c0368195c182ebc173c45172e43940260b"}
Sep 11 18:29:37.775562 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:37.775512    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt" podStartSLOduration=1.775497019 podStartE2EDuration="1.775497019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:37.774693094 +0000 UTC m=+854.515819937" watchObservedRunningTime="2026-09-11 18:29:37.775497019 +0000 UTC m=+854.516623853"
Sep 11 18:29:44.743339 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:44.743306    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"
Sep 11 18:29:44.743730 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:44.743371    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"
Sep 11 18:29:46.941388 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:46.941357    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:46.941388 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:46.941393    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:46.954360 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:46.954338    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:47.822098 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:47.822069    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:29:51.867312 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:29:51.867266    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:30:00.147254 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.147200    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d7509-29819190-jkx86"]
Sep 11 18:30:00.151701 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.151676    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-68ac8212-29819190-xptzn"]
Sep 11 18:30:00.151850 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.151789    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d7509-29819190-jkx86"
Sep 11 18:30:00.154080 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.154027    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\""
Sep 11 18:30:00.155535 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.155506    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-68ac8212-29819190-xptzn"
Sep 11 18:30:00.159286 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.158449    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1de19d-29819190-r24gk"]
Sep 11 18:30:00.163341 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.163121    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d7509-29819190-jkx86"]
Sep 11 18:30:00.163341 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.163213    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1de19d-29819190-r24gk"
Sep 11 18:30:00.165003 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.164984    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-68ac8212-29819190-xptzn"]
Sep 11 18:30:00.167359 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.167338    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d7509-29819190-jkx86"
Sep 11 18:30:00.175366 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.171344    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1de19d-29819190-r24gk"]
Sep 11 18:30:00.179111 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.179090    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-68ac8212-29819190-xptzn"
Sep 11 18:30:00.225296 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.225276    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1de19d-29819190-r24gk"
Sep 11 18:30:00.458304 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.458275    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d7509-29819190-jkx86"]
Sep 11 18:30:00.459811 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:30:00.459790    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928affc0_ec0f_4bbe_8549_39ee5654cbdc.slice/crio-bd244ae1dcc8fb11e9dfbfb4cb108d8044f25bcc50a0b9ddabb1e7df625f8a27 WatchSource:0}: Error finding container bd244ae1dcc8fb11e9dfbfb4cb108d8044f25bcc50a0b9ddabb1e7df625f8a27: Status 404 returned error can't find the container with id bd244ae1dcc8fb11e9dfbfb4cb108d8044f25bcc50a0b9ddabb1e7df625f8a27
Sep 11 18:30:00.460977 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.460956    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-68ac8212-29819190-xptzn"]
Sep 11 18:30:00.461715 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:30:00.461697    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63ae4f1c_ad8c_4d08_bc35_5850ffbc58d8.slice/crio-4d965bcd5dc430799247267f65bf6d284c8741838283f520a70368ca5f88a722 WatchSource:0}: Error finding container 4d965bcd5dc430799247267f65bf6d284c8741838283f520a70368ca5f88a722: Status 404 returned error can't find the container with id 4d965bcd5dc430799247267f65bf6d284c8741838283f520a70368ca5f88a722
Sep 11 18:30:00.515382 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.515356    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1de19d-29819190-r24gk"]
Sep 11 18:30:00.516985 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:30:00.516962    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7430ddfe_7165_4fb9_95fd_1066e668940a.slice/crio-f1bec647877002d5491a1778f2c7a1b022c26af8457a3c55189ff7d28377b33b WatchSource:0}: Error finding container f1bec647877002d5491a1778f2c7a1b022c26af8457a3c55189ff7d28377b33b: Status 404 returned error can't find the container with id f1bec647877002d5491a1778f2c7a1b022c26af8457a3c55189ff7d28377b33b
Sep 11 18:30:00.878119 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.878091    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1de19d-29819190-r24gk" event={"ID":"7430ddfe-7165-4fb9-95fd-1066e668940a","Type":"ContainerStarted","Data":"f1bec647877002d5491a1778f2c7a1b022c26af8457a3c55189ff7d28377b33b"}
Sep 11 18:30:00.879250 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.879226    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-68ac8212-29819190-xptzn" event={"ID":"63ae4f1c-ad8c-4d08-bc35-5850ffbc58d8","Type":"ContainerStarted","Data":"4d965bcd5dc430799247267f65bf6d284c8741838283f520a70368ca5f88a722"}
Sep 11 18:30:00.880399 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:00.880372    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d7509-29819190-jkx86" event={"ID":"928affc0-ec0f-4bbe-8549-39ee5654cbdc","Type":"ContainerStarted","Data":"bd244ae1dcc8fb11e9dfbfb4cb108d8044f25bcc50a0b9ddabb1e7df625f8a27"}
Sep 11 18:30:02.284833 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.284743    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-68ac8212-29819190-xptzn"]
Sep 11 18:30:02.332014 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.331981    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"]
Sep 11 18:30:02.332256 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.332224    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v" podUID="9e7605d7-416c-4ddd-a202-45004a2e446f" containerName="payload-processing" containerID="cri-o://15dd8a649e884db211f5662276a43906f7b4a9c20eb321ff76030cd9b5faf981" gracePeriod=30
Sep 11 18:30:02.337838 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.337421    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"]
Sep 11 18:30:02.337838 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.337652    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd" podUID="2ce93669-4b71-4289-a4c6-52b1660d9d06" containerName="payload-pre-processing" containerID="cri-o://cf91004cdef42735be82b145eef0ac98ed12b81b681c20131fbb6ba147239b95" gracePeriod=30
Sep 11 18:30:02.888513 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.888328    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"
Sep 11 18:30:02.890593 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.890575    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"
Sep 11 18:30:02.894865 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.894845    2581 generic.go:356] "Generic (PLEG): container finished" podID="9e7605d7-416c-4ddd-a202-45004a2e446f" containerID="15dd8a649e884db211f5662276a43906f7b4a9c20eb321ff76030cd9b5faf981" exitCode=0
Sep 11 18:30:02.894985 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.894885    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"
Sep 11 18:30:02.894985 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.894925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v" event={"ID":"9e7605d7-416c-4ddd-a202-45004a2e446f","Type":"ContainerDied","Data":"15dd8a649e884db211f5662276a43906f7b4a9c20eb321ff76030cd9b5faf981"}
Sep 11 18:30:02.894985 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.894979    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v" event={"ID":"9e7605d7-416c-4ddd-a202-45004a2e446f","Type":"ContainerDied","Data":"fbdaf63e691e0ba49e2ca9e5ed6ba3d8730bf192e240598f2bce16e4b1d07639"}
Sep 11 18:30:02.895106 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.895007    2581 scope.go:117] "RemoveContainer" containerID="15dd8a649e884db211f5662276a43906f7b4a9c20eb321ff76030cd9b5faf981"
Sep 11 18:30:02.896276 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.896257    2581 generic.go:356] "Generic (PLEG): container finished" podID="2ce93669-4b71-4289-a4c6-52b1660d9d06" containerID="cf91004cdef42735be82b145eef0ac98ed12b81b681c20131fbb6ba147239b95" exitCode=0
Sep 11 18:30:02.896337 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.896283    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"
Sep 11 18:30:02.896337 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.896315    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd" event={"ID":"2ce93669-4b71-4289-a4c6-52b1660d9d06","Type":"ContainerDied","Data":"cf91004cdef42735be82b145eef0ac98ed12b81b681c20131fbb6ba147239b95"}
Sep 11 18:30:02.896406 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.896345    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd" event={"ID":"2ce93669-4b71-4289-a4c6-52b1660d9d06","Type":"ContainerDied","Data":"49b70ce731e0bcd724a0fbb7745f29d3a1c4a37b47804e68cb401fc3a9cd41f9"}
Sep 11 18:30:02.906793 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.906739    2581 scope.go:117] "RemoveContainer" containerID="15dd8a649e884db211f5662276a43906f7b4a9c20eb321ff76030cd9b5faf981"
Sep 11 18:30:02.907981 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:30:02.907381    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15dd8a649e884db211f5662276a43906f7b4a9c20eb321ff76030cd9b5faf981\": container with ID starting with 15dd8a649e884db211f5662276a43906f7b4a9c20eb321ff76030cd9b5faf981 not found: ID does not exist" containerID="15dd8a649e884db211f5662276a43906f7b4a9c20eb321ff76030cd9b5faf981"
Sep 11 18:30:02.908100 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.907994    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15dd8a649e884db211f5662276a43906f7b4a9c20eb321ff76030cd9b5faf981"} err="failed to get container status \"15dd8a649e884db211f5662276a43906f7b4a9c20eb321ff76030cd9b5faf981\": rpc error: code = NotFound desc = could not find container \"15dd8a649e884db211f5662276a43906f7b4a9c20eb321ff76030cd9b5faf981\": container with ID starting with 15dd8a649e884db211f5662276a43906f7b4a9c20eb321ff76030cd9b5faf981 not found: ID does not exist"
Sep 11 18:30:02.908100 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.908033    2581 scope.go:117] "RemoveContainer" containerID="cf91004cdef42735be82b145eef0ac98ed12b81b681c20131fbb6ba147239b95"
Sep 11 18:30:02.917847 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.917830    2581 scope.go:117] "RemoveContainer" containerID="cf91004cdef42735be82b145eef0ac98ed12b81b681c20131fbb6ba147239b95"
Sep 11 18:30:02.918124 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:30:02.918106    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf91004cdef42735be82b145eef0ac98ed12b81b681c20131fbb6ba147239b95\": container with ID starting with cf91004cdef42735be82b145eef0ac98ed12b81b681c20131fbb6ba147239b95 not found: ID does not exist" containerID="cf91004cdef42735be82b145eef0ac98ed12b81b681c20131fbb6ba147239b95"
Sep 11 18:30:02.918190 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:02.918136    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf91004cdef42735be82b145eef0ac98ed12b81b681c20131fbb6ba147239b95"} err="failed to get container status \"cf91004cdef42735be82b145eef0ac98ed12b81b681c20131fbb6ba147239b95\": rpc error: code = NotFound desc = could not find container \"cf91004cdef42735be82b145eef0ac98ed12b81b681c20131fbb6ba147239b95\": container with ID starting with cf91004cdef42735be82b145eef0ac98ed12b81b681c20131fbb6ba147239b95 not found: ID does not exist"
Sep 11 18:30:03.051504 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.051464    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce93669-4b71-4289-a4c6-52b1660d9d06-plugins-config-volume\") pod \"2ce93669-4b71-4289-a4c6-52b1660d9d06\" (UID: \"2ce93669-4b71-4289-a4c6-52b1660d9d06\") "
Sep 11 18:30:03.051654 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.051558    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7605d7-416c-4ddd-a202-45004a2e446f-plugins-config-volume\") pod \"9e7605d7-416c-4ddd-a202-45004a2e446f\" (UID: \"9e7605d7-416c-4ddd-a202-45004a2e446f\") "
Sep 11 18:30:03.051654 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.051590    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n226d\" (UniqueName: \"kubernetes.io/projected/9e7605d7-416c-4ddd-a202-45004a2e446f-kube-api-access-n226d\") pod \"9e7605d7-416c-4ddd-a202-45004a2e446f\" (UID: \"9e7605d7-416c-4ddd-a202-45004a2e446f\") "
Sep 11 18:30:03.051654 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.051610    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnptq\" (UniqueName: \"kubernetes.io/projected/2ce93669-4b71-4289-a4c6-52b1660d9d06-kube-api-access-fnptq\") pod \"2ce93669-4b71-4289-a4c6-52b1660d9d06\" (UID: \"2ce93669-4b71-4289-a4c6-52b1660d9d06\") "
Sep 11 18:30:03.051902 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.051873    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ce93669-4b71-4289-a4c6-52b1660d9d06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2ce93669-4b71-4289-a4c6-52b1660d9d06" (UID: "2ce93669-4b71-4289-a4c6-52b1660d9d06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:30:03.051979 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.051956    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e7605d7-416c-4ddd-a202-45004a2e446f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e7605d7-416c-4ddd-a202-45004a2e446f" (UID: "9e7605d7-416c-4ddd-a202-45004a2e446f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:30:03.053647 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.053624    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ce93669-4b71-4289-a4c6-52b1660d9d06-kube-api-access-fnptq" (OuterVolumeSpecName: "kube-api-access-fnptq") pod "2ce93669-4b71-4289-a4c6-52b1660d9d06" (UID: "2ce93669-4b71-4289-a4c6-52b1660d9d06"). InnerVolumeSpecName "kube-api-access-fnptq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:30:03.053723 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.053706    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e7605d7-416c-4ddd-a202-45004a2e446f-kube-api-access-n226d" (OuterVolumeSpecName: "kube-api-access-n226d") pod "9e7605d7-416c-4ddd-a202-45004a2e446f" (UID: "9e7605d7-416c-4ddd-a202-45004a2e446f"). InnerVolumeSpecName "kube-api-access-n226d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:30:03.152404 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.152383    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ce93669-4b71-4289-a4c6-52b1660d9d06-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:30:03.152464 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.152409    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7605d7-416c-4ddd-a202-45004a2e446f-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:30:03.152464 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.152419    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n226d\" (UniqueName: \"kubernetes.io/projected/9e7605d7-416c-4ddd-a202-45004a2e446f-kube-api-access-n226d\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:30:03.152464 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.152428    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnptq\" (UniqueName: \"kubernetes.io/projected/2ce93669-4b71-4289-a4c6-52b1660d9d06-kube-api-access-fnptq\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:30:03.221888 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.221860    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"]
Sep 11 18:30:03.223853 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.223835    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-68ac8212-5df966db7c-wlb5v"]
Sep 11 18:30:03.233465 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.233412    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"]
Sep 11 18:30:03.238640 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.238621    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-68ac8212-b48688ff4-xfrhd"]
Sep 11 18:30:03.770146 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.770111    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ce93669-4b71-4289-a4c6-52b1660d9d06" path="/var/lib/kubelet/pods/2ce93669-4b71-4289-a4c6-52b1660d9d06/volumes"
Sep 11 18:30:03.770517 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.770448    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e7605d7-416c-4ddd-a202-45004a2e446f" path="/var/lib/kubelet/pods/9e7605d7-416c-4ddd-a202-45004a2e446f/volumes"
Sep 11 18:30:03.903214 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.903184    2581 generic.go:356] "Generic (PLEG): container finished" podID="928affc0-ec0f-4bbe-8549-39ee5654cbdc" containerID="052084acb90fd9b3e56c80c857a6424bd2167c71e5bac7408028c520fde5b5fa" exitCode=0
Sep 11 18:30:03.903345 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.903214    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d7509-29819190-jkx86" event={"ID":"928affc0-ec0f-4bbe-8549-39ee5654cbdc","Type":"ContainerDied","Data":"052084acb90fd9b3e56c80c857a6424bd2167c71e5bac7408028c520fde5b5fa"}
Sep 11 18:30:03.906621 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.906596    2581 generic.go:356] "Generic (PLEG): container finished" podID="7430ddfe-7165-4fb9-95fd-1066e668940a" containerID="a3037f1df9bc6cf3b824480cb40c51d24fa2e589ffdd58233d4628b73e735be2" exitCode=0
Sep 11 18:30:03.906744 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:03.906628    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1de19d-29819190-r24gk" event={"ID":"7430ddfe-7165-4fb9-95fd-1066e668940a","Type":"ContainerDied","Data":"a3037f1df9bc6cf3b824480cb40c51d24fa2e589ffdd58233d4628b73e735be2"}
Sep 11 18:30:04.913812 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:04.913778    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-68ac8212-29819190-xptzn" event={"ID":"63ae4f1c-ad8c-4d08-bc35-5850ffbc58d8","Type":"ContainerStarted","Data":"9f32c51a639b112b5308e32dc5fc5a6ef6dfc76ffee597cda7f7468ae3a1ec45"}
Sep 11 18:30:04.913812 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:04.913792    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-68ac8212-29819190-xptzn" podUID="63ae4f1c-ad8c-4d08-bc35-5850ffbc58d8" containerName="cleanup" containerID="cri-o://9f32c51a639b112b5308e32dc5fc5a6ef6dfc76ffee597cda7f7468ae3a1ec45" gracePeriod=30
Sep 11 18:30:04.931445 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:04.931390    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-68ac8212-29819190-xptzn" podStartSLOduration=1.11912483 podStartE2EDuration="4.931373195s" totalImagesPullingTime="3.812248365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:04.929393904 +0000 UTC m=+881.670520747" watchObservedRunningTime="2026-09-11 18:30:04.931373195 +0000 UTC m=+881.672500037"
Sep 11 18:30:05.092245 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.092216    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1de19d-29819190-r24gk"
Sep 11 18:30:05.096383 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.096242    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d7509-29819190-jkx86"
Sep 11 18:30:05.178518 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.178426    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-68ac8212-29819190-xptzn"
Sep 11 18:30:05.921318 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.921291    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d7509-29819190-jkx86"
Sep 11 18:30:05.921747 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.921321    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d7509-29819190-jkx86" event={"ID":"928affc0-ec0f-4bbe-8549-39ee5654cbdc","Type":"ContainerDied","Data":"bd244ae1dcc8fb11e9dfbfb4cb108d8044f25bcc50a0b9ddabb1e7df625f8a27"}
Sep 11 18:30:05.921747 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.921364    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd244ae1dcc8fb11e9dfbfb4cb108d8044f25bcc50a0b9ddabb1e7df625f8a27"
Sep 11 18:30:05.922734 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.922714    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1de19d-29819190-r24gk"
Sep 11 18:30:05.922858 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.922755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1de19d-29819190-r24gk" event={"ID":"7430ddfe-7165-4fb9-95fd-1066e668940a","Type":"ContainerDied","Data":"f1bec647877002d5491a1778f2c7a1b022c26af8457a3c55189ff7d28377b33b"}
Sep 11 18:30:05.922858 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.922806    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1bec647877002d5491a1778f2c7a1b022c26af8457a3c55189ff7d28377b33b"
Sep 11 18:30:05.924290 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.924267    2581 generic.go:356] "Generic (PLEG): container finished" podID="63ae4f1c-ad8c-4d08-bc35-5850ffbc58d8" containerID="9f32c51a639b112b5308e32dc5fc5a6ef6dfc76ffee597cda7f7468ae3a1ec45" exitCode=6
Sep 11 18:30:05.924375 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.924296    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-68ac8212-29819190-xptzn"
Sep 11 18:30:05.924375 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.924304    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-68ac8212-29819190-xptzn" event={"ID":"63ae4f1c-ad8c-4d08-bc35-5850ffbc58d8","Type":"ContainerDied","Data":"9f32c51a639b112b5308e32dc5fc5a6ef6dfc76ffee597cda7f7468ae3a1ec45"}
Sep 11 18:30:05.924375 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.924353    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-68ac8212-29819190-xptzn" event={"ID":"63ae4f1c-ad8c-4d08-bc35-5850ffbc58d8","Type":"ContainerDied","Data":"4d965bcd5dc430799247267f65bf6d284c8741838283f520a70368ca5f88a722"}
Sep 11 18:30:05.924510 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.924387    2581 scope.go:117] "RemoveContainer" containerID="9f32c51a639b112b5308e32dc5fc5a6ef6dfc76ffee597cda7f7468ae3a1ec45"
Sep 11 18:30:05.935383 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.935363    2581 scope.go:117] "RemoveContainer" containerID="9f32c51a639b112b5308e32dc5fc5a6ef6dfc76ffee597cda7f7468ae3a1ec45"
Sep 11 18:30:05.935679 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:30:05.935659    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f32c51a639b112b5308e32dc5fc5a6ef6dfc76ffee597cda7f7468ae3a1ec45\": container with ID starting with 9f32c51a639b112b5308e32dc5fc5a6ef6dfc76ffee597cda7f7468ae3a1ec45 not found: ID does not exist" containerID="9f32c51a639b112b5308e32dc5fc5a6ef6dfc76ffee597cda7f7468ae3a1ec45"
Sep 11 18:30:05.935771 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.935684    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f32c51a639b112b5308e32dc5fc5a6ef6dfc76ffee597cda7f7468ae3a1ec45"} err="failed to get container status \"9f32c51a639b112b5308e32dc5fc5a6ef6dfc76ffee597cda7f7468ae3a1ec45\": rpc error: code = NotFound desc = could not find container \"9f32c51a639b112b5308e32dc5fc5a6ef6dfc76ffee597cda7f7468ae3a1ec45\": container with ID starting with 9f32c51a639b112b5308e32dc5fc5a6ef6dfc76ffee597cda7f7468ae3a1ec45 not found: ID does not exist"
Sep 11 18:30:05.946945 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.946923    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-68ac8212-29819190-xptzn"]
Sep 11 18:30:05.950684 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:05.950663    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-68ac8212-29819190-xptzn"]
Sep 11 18:30:07.764401 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:07.764367    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63ae4f1c-ad8c-4d08-bc35-5850ffbc58d8" path="/var/lib/kubelet/pods/63ae4f1c-ad8c-4d08-bc35-5850ffbc58d8/volumes"
Sep 11 18:30:08.544113 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:08.544071    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:30:23.127975 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.127881    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"]
Sep 11 18:30:23.128407 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.128135    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7" podUID="74a69185-1e46-45ec-9b16-448dfa377375" containerName="payload-processing" containerID="cri-o://5cb4ff45f24f605a47740192197c85e84b6c00f6e190da1c3ecd69997ed3cd7a" gracePeriod=30
Sep 11 18:30:23.135493 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.135456    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"]
Sep 11 18:30:23.135936 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.135724    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5" podUID="ad2e2354-3592-4381-b6e4-f80cc20bcb1f" containerName="payload-pre-processing" containerID="cri-o://1ad86bcb997e4049d286697f4b7d00f38e2c30f3178d0ce8833945ac515616e3" gracePeriod=30
Sep 11 18:30:23.418952 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.418925    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"
Sep 11 18:30:23.422121 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.422094    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"
Sep 11 18:30:23.535677 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.535649    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flfn6\" (UniqueName: \"kubernetes.io/projected/ad2e2354-3592-4381-b6e4-f80cc20bcb1f-kube-api-access-flfn6\") pod \"ad2e2354-3592-4381-b6e4-f80cc20bcb1f\" (UID: \"ad2e2354-3592-4381-b6e4-f80cc20bcb1f\") "
Sep 11 18:30:23.535844 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.535711    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a69185-1e46-45ec-9b16-448dfa377375-plugins-config-volume\") pod \"74a69185-1e46-45ec-9b16-448dfa377375\" (UID: \"74a69185-1e46-45ec-9b16-448dfa377375\") "
Sep 11 18:30:23.535844 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.535747    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsb5g\" (UniqueName: \"kubernetes.io/projected/74a69185-1e46-45ec-9b16-448dfa377375-kube-api-access-wsb5g\") pod \"74a69185-1e46-45ec-9b16-448dfa377375\" (UID: \"74a69185-1e46-45ec-9b16-448dfa377375\") "
Sep 11 18:30:23.535844 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.535781    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2e2354-3592-4381-b6e4-f80cc20bcb1f-plugins-config-volume\") pod \"ad2e2354-3592-4381-b6e4-f80cc20bcb1f\" (UID: \"ad2e2354-3592-4381-b6e4-f80cc20bcb1f\") "
Sep 11 18:30:23.536196 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.536162    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a69185-1e46-45ec-9b16-448dfa377375-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74a69185-1e46-45ec-9b16-448dfa377375" (UID: "74a69185-1e46-45ec-9b16-448dfa377375"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:30:23.536301 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.536253    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2e2354-3592-4381-b6e4-f80cc20bcb1f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad2e2354-3592-4381-b6e4-f80cc20bcb1f" (UID: "ad2e2354-3592-4381-b6e4-f80cc20bcb1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:30:23.538053 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.538029    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2e2354-3592-4381-b6e4-f80cc20bcb1f-kube-api-access-flfn6" (OuterVolumeSpecName: "kube-api-access-flfn6") pod "ad2e2354-3592-4381-b6e4-f80cc20bcb1f" (UID: "ad2e2354-3592-4381-b6e4-f80cc20bcb1f"). InnerVolumeSpecName "kube-api-access-flfn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:30:23.538053 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.538037    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a69185-1e46-45ec-9b16-448dfa377375-kube-api-access-wsb5g" (OuterVolumeSpecName: "kube-api-access-wsb5g") pod "74a69185-1e46-45ec-9b16-448dfa377375" (UID: "74a69185-1e46-45ec-9b16-448dfa377375"). InnerVolumeSpecName "kube-api-access-wsb5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:30:23.637399 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.637374    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsb5g\" (UniqueName: \"kubernetes.io/projected/74a69185-1e46-45ec-9b16-448dfa377375-kube-api-access-wsb5g\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:30:23.637399 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.637397    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad2e2354-3592-4381-b6e4-f80cc20bcb1f-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:30:23.637579 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.637406    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flfn6\" (UniqueName: \"kubernetes.io/projected/ad2e2354-3592-4381-b6e4-f80cc20bcb1f-kube-api-access-flfn6\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:30:23.637579 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:23.637415    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74a69185-1e46-45ec-9b16-448dfa377375-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:30:24.031987 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.031899    2581 generic.go:356] "Generic (PLEG): container finished" podID="74a69185-1e46-45ec-9b16-448dfa377375" containerID="5cb4ff45f24f605a47740192197c85e84b6c00f6e190da1c3ecd69997ed3cd7a" exitCode=0
Sep 11 18:30:24.031987 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.031945    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"
Sep 11 18:30:24.032203 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.031984    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7" event={"ID":"74a69185-1e46-45ec-9b16-448dfa377375","Type":"ContainerDied","Data":"5cb4ff45f24f605a47740192197c85e84b6c00f6e190da1c3ecd69997ed3cd7a"}
Sep 11 18:30:24.032203 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.032049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7" event={"ID":"74a69185-1e46-45ec-9b16-448dfa377375","Type":"ContainerDied","Data":"c698940afa18c8b72cdfad428323de18e095f68e31135fe63032c81cf45aafa2"}
Sep 11 18:30:24.032203 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.032083    2581 scope.go:117] "RemoveContainer" containerID="5cb4ff45f24f605a47740192197c85e84b6c00f6e190da1c3ecd69997ed3cd7a"
Sep 11 18:30:24.040879 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.040848    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad2e2354-3592-4381-b6e4-f80cc20bcb1f" containerID="1ad86bcb997e4049d286697f4b7d00f38e2c30f3178d0ce8833945ac515616e3" exitCode=0
Sep 11 18:30:24.041021 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.040901    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"
Sep 11 18:30:24.041021 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.040951    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5" event={"ID":"ad2e2354-3592-4381-b6e4-f80cc20bcb1f","Type":"ContainerDied","Data":"1ad86bcb997e4049d286697f4b7d00f38e2c30f3178d0ce8833945ac515616e3"}
Sep 11 18:30:24.041021 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.041018    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5" event={"ID":"ad2e2354-3592-4381-b6e4-f80cc20bcb1f","Type":"ContainerDied","Data":"8b7f338390239a1cc28c45a843b62c4957cdb1936fd21b2a90c76e6a5b14fd09"}
Sep 11 18:30:24.048655 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.048628    2581 scope.go:117] "RemoveContainer" containerID="5cb4ff45f24f605a47740192197c85e84b6c00f6e190da1c3ecd69997ed3cd7a"
Sep 11 18:30:24.049159 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:30:24.049130    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cb4ff45f24f605a47740192197c85e84b6c00f6e190da1c3ecd69997ed3cd7a\": container with ID starting with 5cb4ff45f24f605a47740192197c85e84b6c00f6e190da1c3ecd69997ed3cd7a not found: ID does not exist" containerID="5cb4ff45f24f605a47740192197c85e84b6c00f6e190da1c3ecd69997ed3cd7a"
Sep 11 18:30:24.049279 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.049168    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cb4ff45f24f605a47740192197c85e84b6c00f6e190da1c3ecd69997ed3cd7a"} err="failed to get container status \"5cb4ff45f24f605a47740192197c85e84b6c00f6e190da1c3ecd69997ed3cd7a\": rpc error: code = NotFound desc = could not find container \"5cb4ff45f24f605a47740192197c85e84b6c00f6e190da1c3ecd69997ed3cd7a\": container with ID starting with 5cb4ff45f24f605a47740192197c85e84b6c00f6e190da1c3ecd69997ed3cd7a not found: ID does not exist"
Sep 11 18:30:24.049279 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.049191    2581 scope.go:117] "RemoveContainer" containerID="1ad86bcb997e4049d286697f4b7d00f38e2c30f3178d0ce8833945ac515616e3"
Sep 11 18:30:24.051521 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.051503    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"]
Sep 11 18:30:24.055583 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.055557    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-7e8ab9-6df74d9fcc-c7mf7"]
Sep 11 18:30:24.057778 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.057762    2581 scope.go:117] "RemoveContainer" containerID="1ad86bcb997e4049d286697f4b7d00f38e2c30f3178d0ce8833945ac515616e3"
Sep 11 18:30:24.058045 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:30:24.058027    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad86bcb997e4049d286697f4b7d00f38e2c30f3178d0ce8833945ac515616e3\": container with ID starting with 1ad86bcb997e4049d286697f4b7d00f38e2c30f3178d0ce8833945ac515616e3 not found: ID does not exist" containerID="1ad86bcb997e4049d286697f4b7d00f38e2c30f3178d0ce8833945ac515616e3"
Sep 11 18:30:24.058094 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.058056    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad86bcb997e4049d286697f4b7d00f38e2c30f3178d0ce8833945ac515616e3"} err="failed to get container status \"1ad86bcb997e4049d286697f4b7d00f38e2c30f3178d0ce8833945ac515616e3\": rpc error: code = NotFound desc = could not find container \"1ad86bcb997e4049d286697f4b7d00f38e2c30f3178d0ce8833945ac515616e3\": container with ID starting with 1ad86bcb997e4049d286697f4b7d00f38e2c30f3178d0ce8833945ac515616e3 not found: ID does not exist"
Sep 11 18:30:24.065514 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.065467    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"]
Sep 11 18:30:24.068015 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:24.067994    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-7e8ab9-bd95565f4-rvcw5"]
Sep 11 18:30:25.763905 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:25.763875    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74a69185-1e46-45ec-9b16-448dfa377375" path="/var/lib/kubelet/pods/74a69185-1e46-45ec-9b16-448dfa377375/volumes"
Sep 11 18:30:25.764255 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:25.764226    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad2e2354-3592-4381-b6e4-f80cc20bcb1f" path="/var/lib/kubelet/pods/ad2e2354-3592-4381-b6e4-f80cc20bcb1f/volumes"
Sep 11 18:30:28.693385 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:28.693345    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"]
Sep 11 18:30:29.444919 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:29.444885    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:30:43.713213 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:43.713172    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt" podUID="e26013c4-2ea1-4906-b4dc-4f27c30dac82" containerName="main" containerID="cri-o://e86026c1839b4b07cb58a3ee572c3ee1f83279f90ef9604065ee942e440c6a97" gracePeriod=45
Sep 11 18:30:43.963442 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:43.963390    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:30:44.123136 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.123104    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26013c4-2ea1-4906-b4dc-4f27c30dac82-tls-certs\") pod \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") "
Sep 11 18:30:44.123287 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.123162    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-dshm\") pod \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") "
Sep 11 18:30:44.123287 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.123184    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-model-cache\") pod \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") "
Sep 11 18:30:44.123287 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.123256    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp8t4\" (UniqueName: \"kubernetes.io/projected/e26013c4-2ea1-4906-b4dc-4f27c30dac82-kube-api-access-rp8t4\") pod \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") "
Sep 11 18:30:44.123427 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.123285    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-home\") pod \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") "
Sep 11 18:30:44.123427 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.123355    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-tmp-dir\") pod \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\" (UID: \"e26013c4-2ea1-4906-b4dc-4f27c30dac82\") "
Sep 11 18:30:44.123565 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.123456    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-model-cache" (OuterVolumeSpecName: "model-cache") pod "e26013c4-2ea1-4906-b4dc-4f27c30dac82" (UID: "e26013c4-2ea1-4906-b4dc-4f27c30dac82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:30:44.123643 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.123620    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-home" (OuterVolumeSpecName: "home") pod "e26013c4-2ea1-4906-b4dc-4f27c30dac82" (UID: "e26013c4-2ea1-4906-b4dc-4f27c30dac82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:30:44.123707 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.123633    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e26013c4-2ea1-4906-b4dc-4f27c30dac82" (UID: "e26013c4-2ea1-4906-b4dc-4f27c30dac82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:30:44.123799 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.123784    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:30:44.123853 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.123805    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:30:44.123853 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.123818    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:30:44.125586 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.125568    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-dshm" (OuterVolumeSpecName: "dshm") pod "e26013c4-2ea1-4906-b4dc-4f27c30dac82" (UID: "e26013c4-2ea1-4906-b4dc-4f27c30dac82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:30:44.125664 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.125641    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e26013c4-2ea1-4906-b4dc-4f27c30dac82-kube-api-access-rp8t4" (OuterVolumeSpecName: "kube-api-access-rp8t4") pod "e26013c4-2ea1-4906-b4dc-4f27c30dac82" (UID: "e26013c4-2ea1-4906-b4dc-4f27c30dac82"). InnerVolumeSpecName "kube-api-access-rp8t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:30:44.125721 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.125707    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e26013c4-2ea1-4906-b4dc-4f27c30dac82-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e26013c4-2ea1-4906-b4dc-4f27c30dac82" (UID: "e26013c4-2ea1-4906-b4dc-4f27c30dac82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:30:44.147727 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.147699    2581 generic.go:356] "Generic (PLEG): container finished" podID="e26013c4-2ea1-4906-b4dc-4f27c30dac82" containerID="e86026c1839b4b07cb58a3ee572c3ee1f83279f90ef9604065ee942e440c6a97" exitCode=0
Sep 11 18:30:44.147842 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.147753    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"
Sep 11 18:30:44.147842 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.147754    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt" event={"ID":"e26013c4-2ea1-4906-b4dc-4f27c30dac82","Type":"ContainerDied","Data":"e86026c1839b4b07cb58a3ee572c3ee1f83279f90ef9604065ee942e440c6a97"}
Sep 11 18:30:44.147955 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.147868    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt" event={"ID":"e26013c4-2ea1-4906-b4dc-4f27c30dac82","Type":"ContainerDied","Data":"6456d156cee0faf7e14d8f5b755465c0368195c182ebc173c45172e43940260b"}
Sep 11 18:30:44.147955 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.147891    2581 scope.go:117] "RemoveContainer" containerID="e86026c1839b4b07cb58a3ee572c3ee1f83279f90ef9604065ee942e440c6a97"
Sep 11 18:30:44.156813 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.156796    2581 scope.go:117] "RemoveContainer" containerID="e86026c1839b4b07cb58a3ee572c3ee1f83279f90ef9604065ee942e440c6a97"
Sep 11 18:30:44.157079 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:30:44.157063    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e86026c1839b4b07cb58a3ee572c3ee1f83279f90ef9604065ee942e440c6a97\": container with ID starting with e86026c1839b4b07cb58a3ee572c3ee1f83279f90ef9604065ee942e440c6a97 not found: ID does not exist" containerID="e86026c1839b4b07cb58a3ee572c3ee1f83279f90ef9604065ee942e440c6a97"
Sep 11 18:30:44.157125 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.157087    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e86026c1839b4b07cb58a3ee572c3ee1f83279f90ef9604065ee942e440c6a97"} err="failed to get container status \"e86026c1839b4b07cb58a3ee572c3ee1f83279f90ef9604065ee942e440c6a97\": rpc error: code = NotFound desc = could not find container \"e86026c1839b4b07cb58a3ee572c3ee1f83279f90ef9604065ee942e440c6a97\": container with ID starting with e86026c1839b4b07cb58a3ee572c3ee1f83279f90ef9604065ee942e440c6a97 not found: ID does not exist"
Sep 11 18:30:44.172068 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.172045    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"]
Sep 11 18:30:44.176513 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.176471    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-7e8ab9/autores-model-7e8ab9-kserve-5ffbd8d4c5-wshqt"]
Sep 11 18:30:44.225074 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.225015    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e26013c4-2ea1-4906-b4dc-4f27c30dac82-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:30:44.225074 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.225039    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp8t4\" (UniqueName: \"kubernetes.io/projected/e26013c4-2ea1-4906-b4dc-4f27c30dac82-kube-api-access-rp8t4\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:30:44.225074 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:44.225053    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e26013c4-2ea1-4906-b4dc-4f27c30dac82-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:30:45.764735 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:30:45.764707    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e26013c4-2ea1-4906-b4dc-4f27c30dac82" path="/var/lib/kubelet/pods/e26013c4-2ea1-4906-b4dc-4f27c30dac82/volumes"
Sep 11 18:31:09.433427 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433393    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"]
Sep 11 18:31:09.433853 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433806    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="928affc0-ec0f-4bbe-8549-39ee5654cbdc" containerName="cleanup"
Sep 11 18:31:09.433853 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433818    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="928affc0-ec0f-4bbe-8549-39ee5654cbdc" containerName="cleanup"
Sep 11 18:31:09.433853 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433827    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7430ddfe-7165-4fb9-95fd-1066e668940a" containerName="cleanup"
Sep 11 18:31:09.433853 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433832    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7430ddfe-7165-4fb9-95fd-1066e668940a" containerName="cleanup"
Sep 11 18:31:09.433853 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433840    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e7605d7-416c-4ddd-a202-45004a2e446f" containerName="payload-processing"
Sep 11 18:31:09.433853 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433846    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e7605d7-416c-4ddd-a202-45004a2e446f" containerName="payload-processing"
Sep 11 18:31:09.433853 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433855    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad2e2354-3592-4381-b6e4-f80cc20bcb1f" containerName="payload-pre-processing"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433860    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad2e2354-3592-4381-b6e4-f80cc20bcb1f" containerName="payload-pre-processing"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433868    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74a69185-1e46-45ec-9b16-448dfa377375" containerName="payload-processing"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433873    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="74a69185-1e46-45ec-9b16-448dfa377375" containerName="payload-processing"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433881    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ce93669-4b71-4289-a4c6-52b1660d9d06" containerName="payload-pre-processing"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433886    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ce93669-4b71-4289-a4c6-52b1660d9d06" containerName="payload-pre-processing"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433891    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63ae4f1c-ad8c-4d08-bc35-5850ffbc58d8" containerName="cleanup"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433896    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="63ae4f1c-ad8c-4d08-bc35-5850ffbc58d8" containerName="cleanup"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433907    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e26013c4-2ea1-4906-b4dc-4f27c30dac82" containerName="main"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433912    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e26013c4-2ea1-4906-b4dc-4f27c30dac82" containerName="main"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433984    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ce93669-4b71-4289-a4c6-52b1660d9d06" containerName="payload-pre-processing"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433991    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="928affc0-ec0f-4bbe-8549-39ee5654cbdc" containerName="cleanup"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.433997    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e26013c4-2ea1-4906-b4dc-4f27c30dac82" containerName="main"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.434003    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad2e2354-3592-4381-b6e4-f80cc20bcb1f" containerName="payload-pre-processing"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.434009    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="63ae4f1c-ad8c-4d08-bc35-5850ffbc58d8" containerName="cleanup"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.434014    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e7605d7-416c-4ddd-a202-45004a2e446f" containerName="payload-processing"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.434021    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="74a69185-1e46-45ec-9b16-448dfa377375" containerName="payload-processing"
Sep 11 18:31:09.434071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.434026    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7430ddfe-7165-4fb9-95fd-1066e668940a" containerName="cleanup"
Sep 11 18:31:09.436656 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.436640    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.438614 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.438594    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-efa8b9\"/\"premium-w0-efa8b9-kserve-self-signed-certs\""
Sep 11 18:31:09.447071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.447047    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"]
Sep 11 18:31:09.548903 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.548875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-tmp-dir\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.549068 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.548914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-home\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.549068 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.548995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-tls-certs\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.549068 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.549019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67xmj\" (UniqueName: \"kubernetes.io/projected/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-kube-api-access-67xmj\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.549068 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.549046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-dshm\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.549068 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.549065    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-model-cache\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.650271 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.650241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-tls-certs\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.650438 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.650274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67xmj\" (UniqueName: \"kubernetes.io/projected/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-kube-api-access-67xmj\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.650438 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.650307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-dshm\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.650438 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.650336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-model-cache\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.650438 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.650413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-tmp-dir\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.650658 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.650448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-home\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.650822 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.650805    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-home\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.650880 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.650846    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-tmp-dir\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.650930 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.650861    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-model-cache\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.652789 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.652766    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-dshm\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.653054 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.653035    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-tls-certs\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.660554 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.660528    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67xmj\" (UniqueName: \"kubernetes.io/projected/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-kube-api-access-67xmj\") pod \"premium-w0-efa8b9-kserve-684d99fdfc-sft2z\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") " pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:09.746844 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:09.746781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:10.019667 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:10.019640    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"]
Sep 11 18:31:10.021993 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:31:10.021969    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc052afe1_73fc_4a08_9299_aa8dea7b8d5e.slice/crio-707a1b85018758219706c14c8f2100a155c1fdb46ae673c74eb538de1b5321dd WatchSource:0}: Error finding container 707a1b85018758219706c14c8f2100a155c1fdb46ae673c74eb538de1b5321dd: Status 404 returned error can't find the container with id 707a1b85018758219706c14c8f2100a155c1fdb46ae673c74eb538de1b5321dd
Sep 11 18:31:10.291238 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:10.291137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z" event={"ID":"c052afe1-73fc-4a08-9299-aa8dea7b8d5e","Type":"ContainerStarted","Data":"64c234f114780acc98ec1212984f2ed6057880be1004a1fcca9152ce0545bea0"}
Sep 11 18:31:10.291238 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:10.291186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z" event={"ID":"c052afe1-73fc-4a08-9299-aa8dea7b8d5e","Type":"ContainerStarted","Data":"707a1b85018758219706c14c8f2100a155c1fdb46ae673c74eb538de1b5321dd"}
Sep 11 18:31:10.312601 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:10.312558    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z" podStartSLOduration=1.312539586 podStartE2EDuration="1.312539586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:10.311378922 +0000 UTC m=+947.052505766" watchObservedRunningTime="2026-09-11 18:31:10.312539586 +0000 UTC m=+947.053666445"
Sep 11 18:31:14.164699 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:14.164660    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:31:15.182886 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:15.182851    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"]
Sep 11 18:31:15.185674 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:15.185657    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"
Sep 11 18:31:15.187580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:15.187551    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c7c83f\""
Sep 11 18:31:15.187724 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:15.187631    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c7c83f-dockercfg-6lvqt\""
Sep 11 18:31:15.198401 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:15.198381    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"]
Sep 11 18:31:15.303779 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:15.303727    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kttj8\" (UniqueName: \"kubernetes.io/projected/56797087-2455-4eb0-be59-77b4deb59bcc-kube-api-access-kttj8\") pod \"payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2\" (UID: \"56797087-2455-4eb0-be59-77b4deb59bcc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"
Sep 11 18:31:15.303779 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:15.303789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56797087-2455-4eb0-be59-77b4deb59bcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2\" (UID: \"56797087-2455-4eb0-be59-77b4deb59bcc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"
Sep 11 18:31:15.405221 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:15.405190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kttj8\" (UniqueName: \"kubernetes.io/projected/56797087-2455-4eb0-be59-77b4deb59bcc-kube-api-access-kttj8\") pod \"payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2\" (UID: \"56797087-2455-4eb0-be59-77b4deb59bcc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"
Sep 11 18:31:15.405221 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:15.405227    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56797087-2455-4eb0-be59-77b4deb59bcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2\" (UID: \"56797087-2455-4eb0-be59-77b4deb59bcc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"
Sep 11 18:31:15.405874 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:15.405853    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56797087-2455-4eb0-be59-77b4deb59bcc-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2\" (UID: \"56797087-2455-4eb0-be59-77b4deb59bcc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"
Sep 11 18:31:15.417936 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:15.417912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kttj8\" (UniqueName: \"kubernetes.io/projected/56797087-2455-4eb0-be59-77b4deb59bcc-kube-api-access-kttj8\") pod \"payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2\" (UID: \"56797087-2455-4eb0-be59-77b4deb59bcc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"
Sep 11 18:31:15.497042 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:15.496955    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"
Sep 11 18:31:15.766814 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:15.766790    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"]
Sep 11 18:31:15.768431 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:31:15.768391    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56797087_2455_4eb0_be59_77b4deb59bcc.slice/crio-a2e0c4299d48ff7c59131e4a53a4a9325ca7cceb3264c7ed33de001702651184 WatchSource:0}: Error finding container a2e0c4299d48ff7c59131e4a53a4a9325ca7cceb3264c7ed33de001702651184: Status 404 returned error can't find the container with id a2e0c4299d48ff7c59131e4a53a4a9325ca7cceb3264c7ed33de001702651184
Sep 11 18:31:16.325348 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:16.325317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2" event={"ID":"56797087-2455-4eb0-be59-77b4deb59bcc","Type":"ContainerStarted","Data":"e0ce5cb24cfd9624be08915fe5e3e91090633243d2517d9c1bd3ad4725731146"}
Sep 11 18:31:16.325736 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:16.325366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2" event={"ID":"56797087-2455-4eb0-be59-77b4deb59bcc","Type":"ContainerStarted","Data":"a2e0c4299d48ff7c59131e4a53a4a9325ca7cceb3264c7ed33de001702651184"}
Sep 11 18:31:16.325736 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:16.325440    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"
Sep 11 18:31:16.341787 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:16.341740    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2" podStartSLOduration=1.34172623 podStartE2EDuration="1.34172623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:16.340040449 +0000 UTC m=+953.081167293" watchObservedRunningTime="2026-09-11 18:31:16.34172623 +0000 UTC m=+953.082853073"
Sep 11 18:31:19.747273 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:19.747224    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:19.747684 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:19.747289    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:19.765019 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:19.764995    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:20.359420 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:20.359391    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:31:27.333243 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:27.333214    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"
Sep 11 18:31:38.444959 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.444924    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"]
Sep 11 18:31:38.448313 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.448292    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.450491 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.450462    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-724327\"/\"default-dockercfg-wrpkw\""
Sep 11 18:31:38.450569 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.450463    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-724327\"/\"kube-root-ca.crt\""
Sep 11 18:31:38.451170 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.451151    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-724327\"/\"openshift-service-ca.crt\""
Sep 11 18:31:38.451231 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.451192    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-724327\"/\"premium-w4-724327-kserve-self-signed-certs\""
Sep 11 18:31:38.458706 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.458684    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"]
Sep 11 18:31:38.623841 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.623809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-tmp-dir\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.623977 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.623854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-dshm\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.623977 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.623955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-home\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.624052 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.623995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87de27b2-081b-4e99-b864-dcd612cc4261-tls-certs\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.624092 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.624073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-model-cache\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.624127 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.624095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkgh6\" (UniqueName: \"kubernetes.io/projected/87de27b2-081b-4e99-b864-dcd612cc4261-kube-api-access-tkgh6\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.725376 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.725309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-home\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.725376 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.725354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87de27b2-081b-4e99-b864-dcd612cc4261-tls-certs\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.725585 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.725382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-model-cache\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.725585 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.725398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkgh6\" (UniqueName: \"kubernetes.io/projected/87de27b2-081b-4e99-b864-dcd612cc4261-kube-api-access-tkgh6\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.725585 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.725420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-tmp-dir\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.725585 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.725444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-dshm\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.725806 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.725779    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-home\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.725874 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.725812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-model-cache\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.725931 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.725894    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-tmp-dir\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.727868 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.727839    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-dshm\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.728051 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.728036    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87de27b2-081b-4e99-b864-dcd612cc4261-tls-certs\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.733690 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.733670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkgh6\" (UniqueName: \"kubernetes.io/projected/87de27b2-081b-4e99-b864-dcd612cc4261-kube-api-access-tkgh6\") pod \"premium-w4-724327-kserve-d68bc4957-xkp5k\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") " pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:38.758422 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:38.758397    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:39.028911 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:39.028807    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"]
Sep 11 18:31:39.030801 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:31:39.030767    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87de27b2_081b_4e99_b864_dcd612cc4261.slice/crio-3b627defe241b169fb621d8af8be1c39545df8e134858d1ed0ab8b72b5e298d7 WatchSource:0}: Error finding container 3b627defe241b169fb621d8af8be1c39545df8e134858d1ed0ab8b72b5e298d7: Status 404 returned error can't find the container with id 3b627defe241b169fb621d8af8be1c39545df8e134858d1ed0ab8b72b5e298d7
Sep 11 18:31:39.448312 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:39.448280    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k" event={"ID":"87de27b2-081b-4e99-b864-dcd612cc4261","Type":"ContainerStarted","Data":"c93a9bc856cd2261b64a54fbbeb10c2b7c4420c30aca037177ffc855213bb1a5"}
Sep 11 18:31:39.448750 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:39.448322    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k" event={"ID":"87de27b2-081b-4e99-b864-dcd612cc4261","Type":"ContainerStarted","Data":"3b627defe241b169fb621d8af8be1c39545df8e134858d1ed0ab8b72b5e298d7"}
Sep 11 18:31:39.466071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:39.466026    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k" podStartSLOduration=1.466011153 podStartE2EDuration="1.466011153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:39.464655373 +0000 UTC m=+976.205782217" watchObservedRunningTime="2026-09-11 18:31:39.466011153 +0000 UTC m=+976.207137995"
Sep 11 18:31:39.554681 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:39.554648    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:31:48.758624 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:48.758530    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:48.758624 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:48.758572    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:48.770972 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:48.770949    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:49.511697 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:49.511670    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:31:49.972005 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:49.971970    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"]
Sep 11 18:31:49.974614 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:49.974597    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"
Sep 11 18:31:49.976519 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:49.976465    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3ad8c202\""
Sep 11 18:31:49.991342 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:49.991320    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"]
Sep 11 18:31:49.998190 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:49.998169    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"]
Sep 11 18:31:50.000762 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.000744    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"
Sep 11 18:31:50.003517 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.003465    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3ad8c202-dockercfg-7vhfn\""
Sep 11 18:31:50.012318 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.012296    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"]
Sep 11 18:31:50.036592 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.036567    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5ce579-037f-45de-a3d8-b4d04a46cd87-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl\" (UID: \"fc5ce579-037f-45de-a3d8-b4d04a46cd87\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"
Sep 11 18:31:50.036690 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.036628    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n4cx\" (UniqueName: \"kubernetes.io/projected/fc5ce579-037f-45de-a3d8-b4d04a46cd87-kube-api-access-4n4cx\") pod \"payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl\" (UID: \"fc5ce579-037f-45de-a3d8-b4d04a46cd87\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"
Sep 11 18:31:50.036750 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.036702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxxdt\" (UniqueName: \"kubernetes.io/projected/56978c58-5297-412d-b30b-a06e1d38db06-kube-api-access-bxxdt\") pod \"payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2\" (UID: \"56978c58-5297-412d-b30b-a06e1d38db06\") " pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"
Sep 11 18:31:50.036750 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.036723    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56978c58-5297-412d-b30b-a06e1d38db06-plugins-config-volume\") pod \"payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2\" (UID: \"56978c58-5297-412d-b30b-a06e1d38db06\") " pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"
Sep 11 18:31:50.137540 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.137455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n4cx\" (UniqueName: \"kubernetes.io/projected/fc5ce579-037f-45de-a3d8-b4d04a46cd87-kube-api-access-4n4cx\") pod \"payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl\" (UID: \"fc5ce579-037f-45de-a3d8-b4d04a46cd87\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"
Sep 11 18:31:50.137698 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.137559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxxdt\" (UniqueName: \"kubernetes.io/projected/56978c58-5297-412d-b30b-a06e1d38db06-kube-api-access-bxxdt\") pod \"payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2\" (UID: \"56978c58-5297-412d-b30b-a06e1d38db06\") " pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"
Sep 11 18:31:50.137698 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.137580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56978c58-5297-412d-b30b-a06e1d38db06-plugins-config-volume\") pod \"payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2\" (UID: \"56978c58-5297-412d-b30b-a06e1d38db06\") " pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"
Sep 11 18:31:50.137698 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.137644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5ce579-037f-45de-a3d8-b4d04a46cd87-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl\" (UID: \"fc5ce579-037f-45de-a3d8-b4d04a46cd87\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"
Sep 11 18:31:50.138297 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.138272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5ce579-037f-45de-a3d8-b4d04a46cd87-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl\" (UID: \"fc5ce579-037f-45de-a3d8-b4d04a46cd87\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"
Sep 11 18:31:50.138399 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.138329    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56978c58-5297-412d-b30b-a06e1d38db06-plugins-config-volume\") pod \"payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2\" (UID: \"56978c58-5297-412d-b30b-a06e1d38db06\") " pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"
Sep 11 18:31:50.150716 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.150684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxxdt\" (UniqueName: \"kubernetes.io/projected/56978c58-5297-412d-b30b-a06e1d38db06-kube-api-access-bxxdt\") pod \"payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2\" (UID: \"56978c58-5297-412d-b30b-a06e1d38db06\") " pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"
Sep 11 18:31:50.151334 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.151310    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n4cx\" (UniqueName: \"kubernetes.io/projected/fc5ce579-037f-45de-a3d8-b4d04a46cd87-kube-api-access-4n4cx\") pod \"payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl\" (UID: \"fc5ce579-037f-45de-a3d8-b4d04a46cd87\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"
Sep 11 18:31:50.285995 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.285907    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"
Sep 11 18:31:50.312195 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.312169    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"
Sep 11 18:31:50.558585 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.558558    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"]
Sep 11 18:31:50.559893 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:31:50.559872    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5ce579_037f_45de_a3d8_b4d04a46cd87.slice/crio-1e05952ac829c05ca5757dc99e2de0a29b87cf7ce267d521655c31f70ff3d120 WatchSource:0}: Error finding container 1e05952ac829c05ca5757dc99e2de0a29b87cf7ce267d521655c31f70ff3d120: Status 404 returned error can't find the container with id 1e05952ac829c05ca5757dc99e2de0a29b87cf7ce267d521655c31f70ff3d120
Sep 11 18:31:50.596595 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:50.596572    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"]
Sep 11 18:31:50.598002 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:31:50.597978    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56978c58_5297_412d_b30b_a06e1d38db06.slice/crio-0f505562de322ee6e64c1e59a6a457def4bd4144fbf0dc4c2bf80f5ee464fdee WatchSource:0}: Error finding container 0f505562de322ee6e64c1e59a6a457def4bd4144fbf0dc4c2bf80f5ee464fdee: Status 404 returned error can't find the container with id 0f505562de322ee6e64c1e59a6a457def4bd4144fbf0dc4c2bf80f5ee464fdee
Sep 11 18:31:51.512247 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:51.512211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2" event={"ID":"56978c58-5297-412d-b30b-a06e1d38db06","Type":"ContainerStarted","Data":"ab48e77f82c8c6db4ec2a406d302d76aff4db61704517c616c0970978772bfc5"}
Sep 11 18:31:51.512729 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:51.512269    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2" event={"ID":"56978c58-5297-412d-b30b-a06e1d38db06","Type":"ContainerStarted","Data":"0f505562de322ee6e64c1e59a6a457def4bd4144fbf0dc4c2bf80f5ee464fdee"}
Sep 11 18:31:51.512729 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:51.512424    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"
Sep 11 18:31:51.513906 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:51.513875    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl" event={"ID":"fc5ce579-037f-45de-a3d8-b4d04a46cd87","Type":"ContainerStarted","Data":"bbd5783374a66e03101f6049cf3d0cc3dd55b4f67002cf6733a4ca8ca123f934"}
Sep 11 18:31:51.514034 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:51.513925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl" event={"ID":"fc5ce579-037f-45de-a3d8-b4d04a46cd87","Type":"ContainerStarted","Data":"1e05952ac829c05ca5757dc99e2de0a29b87cf7ce267d521655c31f70ff3d120"}
Sep 11 18:31:51.514034 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:51.513954    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"
Sep 11 18:31:51.528721 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:51.528678    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2" podStartSLOduration=2.528661687 podStartE2EDuration="2.528661687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:51.526766698 +0000 UTC m=+988.267893553" watchObservedRunningTime="2026-09-11 18:31:51.528661687 +0000 UTC m=+988.269788530"
Sep 11 18:31:51.543008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:31:51.542972    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl" podStartSLOduration=2.542959519 podStartE2EDuration="2.542959519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:51.541258402 +0000 UTC m=+988.282385244" watchObservedRunningTime="2026-09-11 18:31:51.542959519 +0000 UTC m=+988.284086361"
Sep 11 18:32:02.521393 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:02.521360    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"
Sep 11 18:32:02.521878 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:02.521429    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"
Sep 11 18:32:02.950261 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:02.950221    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:32:33.767689 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:33.767659    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"]
Sep 11 18:32:33.768261 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:33.767888    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2" podUID="56978c58-5297-412d-b30b-a06e1d38db06" containerName="payload-processing" containerID="cri-o://ab48e77f82c8c6db4ec2a406d302d76aff4db61704517c616c0970978772bfc5" gracePeriod=30
Sep 11 18:32:33.775207 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:33.775178    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"]
Sep 11 18:32:33.775610 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:33.775560    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl" podUID="fc5ce579-037f-45de-a3d8-b4d04a46cd87" containerName="payload-pre-processing" containerID="cri-o://bbd5783374a66e03101f6049cf3d0cc3dd55b4f67002cf6733a4ca8ca123f934" gracePeriod=30
Sep 11 18:32:34.036861 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.036840    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"
Sep 11 18:32:34.048779 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.048751    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"
Sep 11 18:32:34.164385 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.164352    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56978c58-5297-412d-b30b-a06e1d38db06-plugins-config-volume\") pod \"56978c58-5297-412d-b30b-a06e1d38db06\" (UID: \"56978c58-5297-412d-b30b-a06e1d38db06\") "
Sep 11 18:32:34.164622 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.164434    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5ce579-037f-45de-a3d8-b4d04a46cd87-plugins-config-volume\") pod \"fc5ce579-037f-45de-a3d8-b4d04a46cd87\" (UID: \"fc5ce579-037f-45de-a3d8-b4d04a46cd87\") "
Sep 11 18:32:34.164622 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.164545    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n4cx\" (UniqueName: \"kubernetes.io/projected/fc5ce579-037f-45de-a3d8-b4d04a46cd87-kube-api-access-4n4cx\") pod \"fc5ce579-037f-45de-a3d8-b4d04a46cd87\" (UID: \"fc5ce579-037f-45de-a3d8-b4d04a46cd87\") "
Sep 11 18:32:34.164622 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.164588    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxxdt\" (UniqueName: \"kubernetes.io/projected/56978c58-5297-412d-b30b-a06e1d38db06-kube-api-access-bxxdt\") pod \"56978c58-5297-412d-b30b-a06e1d38db06\" (UID: \"56978c58-5297-412d-b30b-a06e1d38db06\") "
Sep 11 18:32:34.164782 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.164752    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56978c58-5297-412d-b30b-a06e1d38db06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56978c58-5297-412d-b30b-a06e1d38db06" (UID: "56978c58-5297-412d-b30b-a06e1d38db06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:32:34.164851 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.164825    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5ce579-037f-45de-a3d8-b4d04a46cd87-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc5ce579-037f-45de-a3d8-b4d04a46cd87" (UID: "fc5ce579-037f-45de-a3d8-b4d04a46cd87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:32:34.165038 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.165022    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5ce579-037f-45de-a3d8-b4d04a46cd87-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:32:34.165101 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.165041    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56978c58-5297-412d-b30b-a06e1d38db06-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:32:34.166722 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.166698    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5ce579-037f-45de-a3d8-b4d04a46cd87-kube-api-access-4n4cx" (OuterVolumeSpecName: "kube-api-access-4n4cx") pod "fc5ce579-037f-45de-a3d8-b4d04a46cd87" (UID: "fc5ce579-037f-45de-a3d8-b4d04a46cd87"). InnerVolumeSpecName "kube-api-access-4n4cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:32:34.166855 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.166837    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56978c58-5297-412d-b30b-a06e1d38db06-kube-api-access-bxxdt" (OuterVolumeSpecName: "kube-api-access-bxxdt") pod "56978c58-5297-412d-b30b-a06e1d38db06" (UID: "56978c58-5297-412d-b30b-a06e1d38db06"). InnerVolumeSpecName "kube-api-access-bxxdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:32:34.265814 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.265784    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n4cx\" (UniqueName: \"kubernetes.io/projected/fc5ce579-037f-45de-a3d8-b4d04a46cd87-kube-api-access-4n4cx\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:32:34.265814 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.265805    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxxdt\" (UniqueName: \"kubernetes.io/projected/56978c58-5297-412d-b30b-a06e1d38db06-kube-api-access-bxxdt\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:32:34.752418 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.752385    2581 generic.go:356] "Generic (PLEG): container finished" podID="56978c58-5297-412d-b30b-a06e1d38db06" containerID="ab48e77f82c8c6db4ec2a406d302d76aff4db61704517c616c0970978772bfc5" exitCode=0
Sep 11 18:32:34.752630 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.752441    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"
Sep 11 18:32:34.752630 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.752463    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2" event={"ID":"56978c58-5297-412d-b30b-a06e1d38db06","Type":"ContainerDied","Data":"ab48e77f82c8c6db4ec2a406d302d76aff4db61704517c616c0970978772bfc5"}
Sep 11 18:32:34.752630 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.752549    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2" event={"ID":"56978c58-5297-412d-b30b-a06e1d38db06","Type":"ContainerDied","Data":"0f505562de322ee6e64c1e59a6a457def4bd4144fbf0dc4c2bf80f5ee464fdee"}
Sep 11 18:32:34.752630 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.752584    2581 scope.go:117] "RemoveContainer" containerID="ab48e77f82c8c6db4ec2a406d302d76aff4db61704517c616c0970978772bfc5"
Sep 11 18:32:34.754042 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.754018    2581 generic.go:356] "Generic (PLEG): container finished" podID="fc5ce579-037f-45de-a3d8-b4d04a46cd87" containerID="bbd5783374a66e03101f6049cf3d0cc3dd55b4f67002cf6733a4ca8ca123f934" exitCode=0
Sep 11 18:32:34.754138 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.754054    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"
Sep 11 18:32:34.754138 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.754055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl" event={"ID":"fc5ce579-037f-45de-a3d8-b4d04a46cd87","Type":"ContainerDied","Data":"bbd5783374a66e03101f6049cf3d0cc3dd55b4f67002cf6733a4ca8ca123f934"}
Sep 11 18:32:34.754138 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.754109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl" event={"ID":"fc5ce579-037f-45de-a3d8-b4d04a46cd87","Type":"ContainerDied","Data":"1e05952ac829c05ca5757dc99e2de0a29b87cf7ce267d521655c31f70ff3d120"}
Sep 11 18:32:34.762053 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.762035    2581 scope.go:117] "RemoveContainer" containerID="ab48e77f82c8c6db4ec2a406d302d76aff4db61704517c616c0970978772bfc5"
Sep 11 18:32:34.762321 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:32:34.762298    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab48e77f82c8c6db4ec2a406d302d76aff4db61704517c616c0970978772bfc5\": container with ID starting with ab48e77f82c8c6db4ec2a406d302d76aff4db61704517c616c0970978772bfc5 not found: ID does not exist" containerID="ab48e77f82c8c6db4ec2a406d302d76aff4db61704517c616c0970978772bfc5"
Sep 11 18:32:34.762412 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.762327    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab48e77f82c8c6db4ec2a406d302d76aff4db61704517c616c0970978772bfc5"} err="failed to get container status \"ab48e77f82c8c6db4ec2a406d302d76aff4db61704517c616c0970978772bfc5\": rpc error: code = NotFound desc = could not find container \"ab48e77f82c8c6db4ec2a406d302d76aff4db61704517c616c0970978772bfc5\": container with ID starting with ab48e77f82c8c6db4ec2a406d302d76aff4db61704517c616c0970978772bfc5 not found: ID does not exist"
Sep 11 18:32:34.762412 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.762343    2581 scope.go:117] "RemoveContainer" containerID="bbd5783374a66e03101f6049cf3d0cc3dd55b4f67002cf6733a4ca8ca123f934"
Sep 11 18:32:34.772173 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.771954    2581 scope.go:117] "RemoveContainer" containerID="bbd5783374a66e03101f6049cf3d0cc3dd55b4f67002cf6733a4ca8ca123f934"
Sep 11 18:32:34.772390 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:32:34.772228    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd5783374a66e03101f6049cf3d0cc3dd55b4f67002cf6733a4ca8ca123f934\": container with ID starting with bbd5783374a66e03101f6049cf3d0cc3dd55b4f67002cf6733a4ca8ca123f934 not found: ID does not exist" containerID="bbd5783374a66e03101f6049cf3d0cc3dd55b4f67002cf6733a4ca8ca123f934"
Sep 11 18:32:34.772390 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.772250    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd5783374a66e03101f6049cf3d0cc3dd55b4f67002cf6733a4ca8ca123f934"} err="failed to get container status \"bbd5783374a66e03101f6049cf3d0cc3dd55b4f67002cf6733a4ca8ca123f934\": rpc error: code = NotFound desc = could not find container \"bbd5783374a66e03101f6049cf3d0cc3dd55b4f67002cf6733a4ca8ca123f934\": container with ID starting with bbd5783374a66e03101f6049cf3d0cc3dd55b4f67002cf6733a4ca8ca123f934 not found: ID does not exist"
Sep 11 18:32:34.779668 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.779643    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"]
Sep 11 18:32:34.782107 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.782086    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3ad8c202-5855ff9857-vhcb2"]
Sep 11 18:32:34.792759 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.792739    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"]
Sep 11 18:32:34.799225 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:34.799206    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3ad8c202-79c6874bbb-5mlzl"]
Sep 11 18:32:35.766595 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:35.766567    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56978c58-5297-412d-b30b-a06e1d38db06" path="/var/lib/kubelet/pods/56978c58-5297-412d-b30b-a06e1d38db06/volumes"
Sep 11 18:32:35.766962 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:35.766948    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc5ce579-037f-45de-a3d8-b4d04a46cd87" path="/var/lib/kubelet/pods/fc5ce579-037f-45de-a3d8-b4d04a46cd87/volumes"
Sep 11 18:32:38.545904 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:38.545869    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:32:43.632780 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.632747    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"]
Sep 11 18:32:43.633373 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.633356    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56978c58-5297-412d-b30b-a06e1d38db06" containerName="payload-processing"
Sep 11 18:32:43.633423 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.633376    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="56978c58-5297-412d-b30b-a06e1d38db06" containerName="payload-processing"
Sep 11 18:32:43.633423 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.633388    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc5ce579-037f-45de-a3d8-b4d04a46cd87" containerName="payload-pre-processing"
Sep 11 18:32:43.633423 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.633394    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc5ce579-037f-45de-a3d8-b4d04a46cd87" containerName="payload-pre-processing"
Sep 11 18:32:43.633579 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.633463    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc5ce579-037f-45de-a3d8-b4d04a46cd87" containerName="payload-pre-processing"
Sep 11 18:32:43.633579 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.633490    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="56978c58-5297-412d-b30b-a06e1d38db06" containerName="payload-processing"
Sep 11 18:32:43.636937 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.636911    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.640026 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.639997    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c7c83f\"/\"explicit-model-c7c83f-kserve-self-signed-certs\""
Sep 11 18:32:43.640150 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.639999    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c7c83f\"/\"default-dockercfg-vhw4n\""
Sep 11 18:32:43.640150 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.640069    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c7c83f\"/\"openshift-service-ca.crt\""
Sep 11 18:32:43.640150 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.639997    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c7c83f\"/\"kube-root-ca.crt\""
Sep 11 18:32:43.646897 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.646877    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"]
Sep 11 18:32:43.759682 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.759638    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-home\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.759682 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.759682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-model-cache\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.759945 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.759771    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-dshm\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.759945 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.759796    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-tmp-dir\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.759945 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.759832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-475b6\" (UniqueName: \"kubernetes.io/projected/a416402a-d9bc-4dfd-81a5-c27eb01df21d-kube-api-access-475b6\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.759945 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.759863    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a416402a-d9bc-4dfd-81a5-c27eb01df21d-tls-certs\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.861260 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.861229    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-dshm\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.861260 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.861265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-tmp-dir\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.861509 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.861306    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-475b6\" (UniqueName: \"kubernetes.io/projected/a416402a-d9bc-4dfd-81a5-c27eb01df21d-kube-api-access-475b6\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.861509 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.861332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a416402a-d9bc-4dfd-81a5-c27eb01df21d-tls-certs\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.861509 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.861389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-home\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.861509 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.861404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-model-cache\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.861756 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.861732    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-tmp-dir\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.861816 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.861795    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-model-cache\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.861854 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.861804    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-home\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.863717 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.863693    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-dshm\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.863929 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.863912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a416402a-d9bc-4dfd-81a5-c27eb01df21d-tls-certs\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.872984 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.872952    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-475b6\" (UniqueName: \"kubernetes.io/projected/a416402a-d9bc-4dfd-81a5-c27eb01df21d-kube-api-access-475b6\") pod \"explicit-model-c7c83f-kserve-6d746d89fc-svkvh\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") " pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:43.946166 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:43.946088    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:44.223748 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:44.223724    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"]
Sep 11 18:32:44.225796 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:32:44.225768    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda416402a_d9bc_4dfd_81a5_c27eb01df21d.slice/crio-3469c2077df2c3661363ac3ad02800c89b42496fbcc7919bf96a65e9999747e1 WatchSource:0}: Error finding container 3469c2077df2c3661363ac3ad02800c89b42496fbcc7919bf96a65e9999747e1: Status 404 returned error can't find the container with id 3469c2077df2c3661363ac3ad02800c89b42496fbcc7919bf96a65e9999747e1
Sep 11 18:32:44.813065 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:44.813028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh" event={"ID":"a416402a-d9bc-4dfd-81a5-c27eb01df21d","Type":"ContainerStarted","Data":"4c9322e946dd3f38beb0ecea53ff92309b2e83251c5888fe33730f8d18d6bd54"}
Sep 11 18:32:44.813441 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:44.813077    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh" event={"ID":"a416402a-d9bc-4dfd-81a5-c27eb01df21d","Type":"ContainerStarted","Data":"3469c2077df2c3661363ac3ad02800c89b42496fbcc7919bf96a65e9999747e1"}
Sep 11 18:32:44.831606 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:44.831564    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh" podStartSLOduration=1.831547741 podStartE2EDuration="1.831547741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:32:44.830400743 +0000 UTC m=+1041.571527604" watchObservedRunningTime="2026-09-11 18:32:44.831547741 +0000 UTC m=+1041.572674583"
Sep 11 18:32:53.946696 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:53.946665    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:53.947144 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:53.946737    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:53.961828 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:53.961801    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:32:54.876741 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:32:54.876712    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:33:03.336396 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.336316    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"]
Sep 11 18:33:03.340161 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.340143    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.342373 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.342350    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-724327\"/\"e2e-conflict-a-498c8da2-kserve-self-signed-certs\""
Sep 11 18:33:03.350677 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.350656    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"]
Sep 11 18:33:03.453299 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.453267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-tmp-dir\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.453456 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.453337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-home\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.453456 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.453363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-model-cache\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.453456 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.453381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38281df6-6cb3-4859-bb54-dfd57447d45b-tls-certs\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.453456 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.453432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9ftm\" (UniqueName: \"kubernetes.io/projected/38281df6-6cb3-4859-bb54-dfd57447d45b-kube-api-access-d9ftm\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.453625 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.453497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-dshm\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.554406 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.554376    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-tmp-dir\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.554571 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.554425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-home\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.554571 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.554462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-model-cache\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.554571 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.554505    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38281df6-6cb3-4859-bb54-dfd57447d45b-tls-certs\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.554708 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.554636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9ftm\" (UniqueName: \"kubernetes.io/projected/38281df6-6cb3-4859-bb54-dfd57447d45b-kube-api-access-d9ftm\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.554708 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.554686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-dshm\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.554810 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.554798    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-tmp-dir\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.554859 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.554825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-model-cache\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.554914 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.554877    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-home\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.556861 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.556840    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-dshm\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.557017 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.557002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38281df6-6cb3-4859-bb54-dfd57447d45b-tls-certs\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.562924 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.562904    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9ftm\" (UniqueName: \"kubernetes.io/projected/38281df6-6cb3-4859-bb54-dfd57447d45b-kube-api-access-d9ftm\") pod \"e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") " pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.650762 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.650727    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:03.923070 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:03.923043    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"]
Sep 11 18:33:03.924498 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:33:03.924455    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38281df6_6cb3_4859_bb54_dfd57447d45b.slice/crio-21e965cb2d0c545af6776ffc0ccf7e4f69c34c6de4fd62c084dfe15e65fe28e3 WatchSource:0}: Error finding container 21e965cb2d0c545af6776ffc0ccf7e4f69c34c6de4fd62c084dfe15e65fe28e3: Status 404 returned error can't find the container with id 21e965cb2d0c545af6776ffc0ccf7e4f69c34c6de4fd62c084dfe15e65fe28e3
Sep 11 18:33:04.711267 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:04.711230    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"]
Sep 11 18:33:04.714822 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:04.714805    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"
Sep 11 18:33:04.717101 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:04.717078    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-869176\""
Sep 11 18:33:04.726106 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:04.726086    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"]
Sep 11 18:33:04.766626 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:04.766600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkfqz\" (UniqueName: \"kubernetes.io/projected/c225c98b-9c37-4d3b-9548-3819ae93289c-kube-api-access-vkfqz\") pod \"payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk\" (UID: \"c225c98b-9c37-4d3b-9548-3819ae93289c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"
Sep 11 18:33:04.767077 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:04.766633    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c225c98b-9c37-4d3b-9548-3819ae93289c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk\" (UID: \"c225c98b-9c37-4d3b-9548-3819ae93289c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"
Sep 11 18:33:04.874203 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:04.874161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkfqz\" (UniqueName: \"kubernetes.io/projected/c225c98b-9c37-4d3b-9548-3819ae93289c-kube-api-access-vkfqz\") pod \"payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk\" (UID: \"c225c98b-9c37-4d3b-9548-3819ae93289c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"
Sep 11 18:33:04.874203 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:04.874196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c225c98b-9c37-4d3b-9548-3819ae93289c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk\" (UID: \"c225c98b-9c37-4d3b-9548-3819ae93289c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"
Sep 11 18:33:04.874936 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:04.874915    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c225c98b-9c37-4d3b-9548-3819ae93289c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk\" (UID: \"c225c98b-9c37-4d3b-9548-3819ae93289c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"
Sep 11 18:33:04.882354 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:04.882334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkfqz\" (UniqueName: \"kubernetes.io/projected/c225c98b-9c37-4d3b-9548-3819ae93289c-kube-api-access-vkfqz\") pod \"payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk\" (UID: \"c225c98b-9c37-4d3b-9548-3819ae93289c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"
Sep 11 18:33:04.920187 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:04.920152    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb" event={"ID":"38281df6-6cb3-4859-bb54-dfd57447d45b","Type":"ContainerStarted","Data":"886b9bf18d47c73b22f3da90dc1c469fc12d4cbe0a85240881e8ef59f98cdbac"}
Sep 11 18:33:04.920287 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:04.920203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb" event={"ID":"38281df6-6cb3-4859-bb54-dfd57447d45b","Type":"ContainerStarted","Data":"21e965cb2d0c545af6776ffc0ccf7e4f69c34c6de4fd62c084dfe15e65fe28e3"}
Sep 11 18:33:04.937217 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:04.937176    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb" podStartSLOduration=1.9371634439999998 podStartE2EDuration="1.937163444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:04.935118879 +0000 UTC m=+1061.676245748" watchObservedRunningTime="2026-09-11 18:33:04.937163444 +0000 UTC m=+1061.678290287"
Sep 11 18:33:05.026939 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:05.026879    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"
Sep 11 18:33:05.297301 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:05.297277    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"]
Sep 11 18:33:05.299505 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:33:05.299456    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc225c98b_9c37_4d3b_9548_3819ae93289c.slice/crio-26389c483aaa4642b5dbfe092ea2dc48c13f65c5a55b8733285e3aab67ddd8e5 WatchSource:0}: Error finding container 26389c483aaa4642b5dbfe092ea2dc48c13f65c5a55b8733285e3aab67ddd8e5: Status 404 returned error can't find the container with id 26389c483aaa4642b5dbfe092ea2dc48c13f65c5a55b8733285e3aab67ddd8e5
Sep 11 18:33:05.927563 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:05.927526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk" event={"ID":"c225c98b-9c37-4d3b-9548-3819ae93289c","Type":"ContainerStarted","Data":"23d5dc1810743258683c6cc0c514778b5f0a806103b4fad6b872b7861a2288be"}
Sep 11 18:33:05.928122 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:05.927589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk" event={"ID":"c225c98b-9c37-4d3b-9548-3819ae93289c","Type":"ContainerStarted","Data":"26389c483aaa4642b5dbfe092ea2dc48c13f65c5a55b8733285e3aab67ddd8e5"}
Sep 11 18:33:05.928122 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:05.927728    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"
Sep 11 18:33:05.946519 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:05.946455    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk" podStartSLOduration=1.946441324 podStartE2EDuration="1.946441324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:05.944634091 +0000 UTC m=+1062.685760935" watchObservedRunningTime="2026-09-11 18:33:05.946441324 +0000 UTC m=+1062.687568175"
Sep 11 18:33:08.832140 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:08.832110    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"]
Sep 11 18:33:12.847100 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:12.847065    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:33:13.651167 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:13.651143    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:13.651343 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:13.651187    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:13.663455 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:13.663431    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:13.987314 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:13.987225    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:33:16.936040 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:16.935969    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"
Sep 11 18:33:21.560063 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:21.560031    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"]
Sep 11 18:33:21.560565 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:21.560256    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2" podUID="56797087-2455-4eb0-be59-77b4deb59bcc" containerName="payload-pre-processing" containerID="cri-o://e0ce5cb24cfd9624be08915fe5e3e91090633243d2517d9c1bd3ad4725731146" gracePeriod=30
Sep 11 18:33:21.826124 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:21.826066    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"
Sep 11 18:33:21.945819 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:21.945786    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kttj8\" (UniqueName: \"kubernetes.io/projected/56797087-2455-4eb0-be59-77b4deb59bcc-kube-api-access-kttj8\") pod \"56797087-2455-4eb0-be59-77b4deb59bcc\" (UID: \"56797087-2455-4eb0-be59-77b4deb59bcc\") "
Sep 11 18:33:21.946008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:21.945873    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56797087-2455-4eb0-be59-77b4deb59bcc-plugins-config-volume\") pod \"56797087-2455-4eb0-be59-77b4deb59bcc\" (UID: \"56797087-2455-4eb0-be59-77b4deb59bcc\") "
Sep 11 18:33:21.946310 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:21.946272    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56797087-2455-4eb0-be59-77b4deb59bcc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56797087-2455-4eb0-be59-77b4deb59bcc" (UID: "56797087-2455-4eb0-be59-77b4deb59bcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:33:21.948080 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:21.948060    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56797087-2455-4eb0-be59-77b4deb59bcc-kube-api-access-kttj8" (OuterVolumeSpecName: "kube-api-access-kttj8") pod "56797087-2455-4eb0-be59-77b4deb59bcc" (UID: "56797087-2455-4eb0-be59-77b4deb59bcc"). InnerVolumeSpecName "kube-api-access-kttj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:33:22.026286 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:22.026250    2581 generic.go:356] "Generic (PLEG): container finished" podID="56797087-2455-4eb0-be59-77b4deb59bcc" containerID="e0ce5cb24cfd9624be08915fe5e3e91090633243d2517d9c1bd3ad4725731146" exitCode=0
Sep 11 18:33:22.026286 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:22.026293    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"
Sep 11 18:33:22.026492 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:22.026329    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2" event={"ID":"56797087-2455-4eb0-be59-77b4deb59bcc","Type":"ContainerDied","Data":"e0ce5cb24cfd9624be08915fe5e3e91090633243d2517d9c1bd3ad4725731146"}
Sep 11 18:33:22.026492 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:22.026388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2" event={"ID":"56797087-2455-4eb0-be59-77b4deb59bcc","Type":"ContainerDied","Data":"a2e0c4299d48ff7c59131e4a53a4a9325ca7cceb3264c7ed33de001702651184"}
Sep 11 18:33:22.026492 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:22.026409    2581 scope.go:117] "RemoveContainer" containerID="e0ce5cb24cfd9624be08915fe5e3e91090633243d2517d9c1bd3ad4725731146"
Sep 11 18:33:22.035575 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:22.035553    2581 scope.go:117] "RemoveContainer" containerID="e0ce5cb24cfd9624be08915fe5e3e91090633243d2517d9c1bd3ad4725731146"
Sep 11 18:33:22.035815 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:33:22.035797    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0ce5cb24cfd9624be08915fe5e3e91090633243d2517d9c1bd3ad4725731146\": container with ID starting with e0ce5cb24cfd9624be08915fe5e3e91090633243d2517d9c1bd3ad4725731146 not found: ID does not exist" containerID="e0ce5cb24cfd9624be08915fe5e3e91090633243d2517d9c1bd3ad4725731146"
Sep 11 18:33:22.035864 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:22.035822    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0ce5cb24cfd9624be08915fe5e3e91090633243d2517d9c1bd3ad4725731146"} err="failed to get container status \"e0ce5cb24cfd9624be08915fe5e3e91090633243d2517d9c1bd3ad4725731146\": rpc error: code = NotFound desc = could not find container \"e0ce5cb24cfd9624be08915fe5e3e91090633243d2517d9c1bd3ad4725731146\": container with ID starting with e0ce5cb24cfd9624be08915fe5e3e91090633243d2517d9c1bd3ad4725731146 not found: ID does not exist"
Sep 11 18:33:22.047067 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:22.047045    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kttj8\" (UniqueName: \"kubernetes.io/projected/56797087-2455-4eb0-be59-77b4deb59bcc-kube-api-access-kttj8\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:33:22.047067 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:22.047068    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56797087-2455-4eb0-be59-77b4deb59bcc-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:33:22.048118 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:22.048097    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"]
Sep 11 18:33:22.055547 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:22.055528    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c7c83f-5468f6f98-75wh2"]
Sep 11 18:33:23.765461 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:23.765430    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56797087-2455-4eb0-be59-77b4deb59bcc" path="/var/lib/kubelet/pods/56797087-2455-4eb0-be59-77b4deb59bcc/volumes"
Sep 11 18:33:23.852268 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:23.852239    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh" podUID="a416402a-d9bc-4dfd-81a5-c27eb01df21d" containerName="main" containerID="cri-o://4c9322e946dd3f38beb0ecea53ff92309b2e83251c5888fe33730f8d18d6bd54" gracePeriod=45
Sep 11 18:33:24.043537 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.043373    2581 generic.go:356] "Generic (PLEG): container finished" podID="a416402a-d9bc-4dfd-81a5-c27eb01df21d" containerID="4c9322e946dd3f38beb0ecea53ff92309b2e83251c5888fe33730f8d18d6bd54" exitCode=0
Sep 11 18:33:24.043537 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.043434    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh" event={"ID":"a416402a-d9bc-4dfd-81a5-c27eb01df21d","Type":"ContainerDied","Data":"4c9322e946dd3f38beb0ecea53ff92309b2e83251c5888fe33730f8d18d6bd54"}
Sep 11 18:33:24.118402 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.118219    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:33:24.268851 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.268766    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-dshm\") pod \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") "
Sep 11 18:33:24.268851 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.268802    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-home\") pod \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") "
Sep 11 18:33:24.268851 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.268843    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a416402a-d9bc-4dfd-81a5-c27eb01df21d-tls-certs\") pod \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") "
Sep 11 18:33:24.269126 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.268880    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-475b6\" (UniqueName: \"kubernetes.io/projected/a416402a-d9bc-4dfd-81a5-c27eb01df21d-kube-api-access-475b6\") pod \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") "
Sep 11 18:33:24.269126 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.268907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-model-cache\") pod \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") "
Sep 11 18:33:24.269126 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.268980    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-tmp-dir\") pod \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\" (UID: \"a416402a-d9bc-4dfd-81a5-c27eb01df21d\") "
Sep 11 18:33:24.269283 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.269117    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-home" (OuterVolumeSpecName: "home") pod "a416402a-d9bc-4dfd-81a5-c27eb01df21d" (UID: "a416402a-d9bc-4dfd-81a5-c27eb01df21d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:33:24.269283 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.269199    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-model-cache" (OuterVolumeSpecName: "model-cache") pod "a416402a-d9bc-4dfd-81a5-c27eb01df21d" (UID: "a416402a-d9bc-4dfd-81a5-c27eb01df21d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:33:24.269283 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.269227    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a416402a-d9bc-4dfd-81a5-c27eb01df21d" (UID: "a416402a-d9bc-4dfd-81a5-c27eb01df21d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:33:24.269444 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.269311    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:33:24.269444 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.269323    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:33:24.269444 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.269331    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:33:24.271193 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.271171    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-dshm" (OuterVolumeSpecName: "dshm") pod "a416402a-d9bc-4dfd-81a5-c27eb01df21d" (UID: "a416402a-d9bc-4dfd-81a5-c27eb01df21d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:33:24.271296 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.271224    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a416402a-d9bc-4dfd-81a5-c27eb01df21d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a416402a-d9bc-4dfd-81a5-c27eb01df21d" (UID: "a416402a-d9bc-4dfd-81a5-c27eb01df21d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:33:24.271296 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.271235    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a416402a-d9bc-4dfd-81a5-c27eb01df21d-kube-api-access-475b6" (OuterVolumeSpecName: "kube-api-access-475b6") pod "a416402a-d9bc-4dfd-81a5-c27eb01df21d" (UID: "a416402a-d9bc-4dfd-81a5-c27eb01df21d"). InnerVolumeSpecName "kube-api-access-475b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:33:24.370461 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.370436    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a416402a-d9bc-4dfd-81a5-c27eb01df21d-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:33:24.370461 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.370459    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-475b6\" (UniqueName: \"kubernetes.io/projected/a416402a-d9bc-4dfd-81a5-c27eb01df21d-kube-api-access-475b6\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:33:24.370633 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:24.370469    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a416402a-d9bc-4dfd-81a5-c27eb01df21d-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:33:25.050170 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:25.050133    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh" event={"ID":"a416402a-d9bc-4dfd-81a5-c27eb01df21d","Type":"ContainerDied","Data":"3469c2077df2c3661363ac3ad02800c89b42496fbcc7919bf96a65e9999747e1"}
Sep 11 18:33:25.050634 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:25.050188    2581 scope.go:117] "RemoveContainer" containerID="4c9322e946dd3f38beb0ecea53ff92309b2e83251c5888fe33730f8d18d6bd54"
Sep 11 18:33:25.050634 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:25.050145    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"
Sep 11 18:33:25.077607 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:25.077580    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"]
Sep 11 18:33:25.082126 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:25.082105    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c7c83f/explicit-model-c7c83f-kserve-6d746d89fc-svkvh"]
Sep 11 18:33:25.765935 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:25.765899    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a416402a-d9bc-4dfd-81a5-c27eb01df21d" path="/var/lib/kubelet/pods/a416402a-d9bc-4dfd-81a5-c27eb01df21d/volumes"
Sep 11 18:33:28.148537 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.148497    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"]
Sep 11 18:33:28.148952 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.148939    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56797087-2455-4eb0-be59-77b4deb59bcc" containerName="payload-pre-processing"
Sep 11 18:33:28.149001 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.148953    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="56797087-2455-4eb0-be59-77b4deb59bcc" containerName="payload-pre-processing"
Sep 11 18:33:28.149001 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.148971    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a416402a-d9bc-4dfd-81a5-c27eb01df21d" containerName="main"
Sep 11 18:33:28.149001 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.148976    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a416402a-d9bc-4dfd-81a5-c27eb01df21d" containerName="main"
Sep 11 18:33:28.149097 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.149037    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="56797087-2455-4eb0-be59-77b4deb59bcc" containerName="payload-pre-processing"
Sep 11 18:33:28.149097 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.149045    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a416402a-d9bc-4dfd-81a5-c27eb01df21d" containerName="main"
Sep 11 18:33:28.153600 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.153581    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"
Sep 11 18:33:28.155549 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.155527    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-0ead89-dockercfg-t89xd\""
Sep 11 18:33:28.155628 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.155535    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-0ead89\""
Sep 11 18:33:28.163860 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.163839    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"]
Sep 11 18:33:28.209302 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.209275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9swnv\" (UniqueName: \"kubernetes.io/projected/6b1e464f-bf95-4b38-8112-f5003d21c234-kube-api-access-9swnv\") pod \"payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76\" (UID: \"6b1e464f-bf95-4b38-8112-f5003d21c234\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"
Sep 11 18:33:28.209409 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.209314    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b1e464f-bf95-4b38-8112-f5003d21c234-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76\" (UID: \"6b1e464f-bf95-4b38-8112-f5003d21c234\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"
Sep 11 18:33:28.310646 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.310609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9swnv\" (UniqueName: \"kubernetes.io/projected/6b1e464f-bf95-4b38-8112-f5003d21c234-kube-api-access-9swnv\") pod \"payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76\" (UID: \"6b1e464f-bf95-4b38-8112-f5003d21c234\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"
Sep 11 18:33:28.310777 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.310659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b1e464f-bf95-4b38-8112-f5003d21c234-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76\" (UID: \"6b1e464f-bf95-4b38-8112-f5003d21c234\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"
Sep 11 18:33:28.311293 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.311276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b1e464f-bf95-4b38-8112-f5003d21c234-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76\" (UID: \"6b1e464f-bf95-4b38-8112-f5003d21c234\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"
Sep 11 18:33:28.319372 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.319353    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9swnv\" (UniqueName: \"kubernetes.io/projected/6b1e464f-bf95-4b38-8112-f5003d21c234-kube-api-access-9swnv\") pod \"payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76\" (UID: \"6b1e464f-bf95-4b38-8112-f5003d21c234\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"
Sep 11 18:33:28.464097 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.464033    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"
Sep 11 18:33:28.737867 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:28.737841    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"]
Sep 11 18:33:28.739833 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:33:28.739807    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1e464f_bf95_4b38_8112_f5003d21c234.slice/crio-e3e95da149191a41b4a9d6ab2587ff83dc64655253789779a1d1688e8352dafb WatchSource:0}: Error finding container e3e95da149191a41b4a9d6ab2587ff83dc64655253789779a1d1688e8352dafb: Status 404 returned error can't find the container with id e3e95da149191a41b4a9d6ab2587ff83dc64655253789779a1d1688e8352dafb
Sep 11 18:33:29.082319 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:29.082212    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76" event={"ID":"6b1e464f-bf95-4b38-8112-f5003d21c234","Type":"ContainerStarted","Data":"6fa6fa2eb25488cfbd53a3d9a996ab9b47fa0a3030b7112f5329bdc612ccdcac"}
Sep 11 18:33:29.082319 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:29.082269    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76" event={"ID":"6b1e464f-bf95-4b38-8112-f5003d21c234","Type":"ContainerStarted","Data":"e3e95da149191a41b4a9d6ab2587ff83dc64655253789779a1d1688e8352dafb"}
Sep 11 18:33:29.082319 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:29.082293    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"
Sep 11 18:33:29.098251 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:29.098199    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76" podStartSLOduration=1.098182086 podStartE2EDuration="1.098182086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:29.097551916 +0000 UTC m=+1085.838678759" watchObservedRunningTime="2026-09-11 18:33:29.098182086 +0000 UTC m=+1085.839308926"
Sep 11 18:33:40.090038 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:40.090009    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"
Sep 11 18:33:59.528615 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.528582    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"]
Sep 11 18:33:59.532350 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.532329    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.534508 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.534464    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-efa8b9\"/\"distinct-w0-efa8b9-kserve-self-signed-certs\""
Sep 11 18:33:59.540580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.540559    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"]
Sep 11 18:33:59.609219 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.609190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pnp6\" (UniqueName: \"kubernetes.io/projected/31321600-870d-4814-ab30-121514d58e5d-kube-api-access-2pnp6\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.609373 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.609234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-dshm\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.609373 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.609316    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31321600-870d-4814-ab30-121514d58e5d-tls-certs\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.609466 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.609429    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-model-cache\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.609529 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.609462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-tmp-dir\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.609529 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.609517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-home\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.710292 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.710260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pnp6\" (UniqueName: \"kubernetes.io/projected/31321600-870d-4814-ab30-121514d58e5d-kube-api-access-2pnp6\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.710292 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.710297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-dshm\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.710536 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.710325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31321600-870d-4814-ab30-121514d58e5d-tls-certs\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.710536 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.710375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-model-cache\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.710536 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.710394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-tmp-dir\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.710536 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.710407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-home\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.710830 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.710796    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-home\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.710830 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.710816    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-tmp-dir\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.711175 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.710873    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-model-cache\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.712627 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.712609    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-dshm\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.713050 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.713032    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31321600-870d-4814-ab30-121514d58e5d-tls-certs\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.719628 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.719607    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pnp6\" (UniqueName: \"kubernetes.io/projected/31321600-870d-4814-ab30-121514d58e5d-kube-api-access-2pnp6\") pod \"distinct-w0-efa8b9-kserve-56c66d4c8d-695xh\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") " pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:33:59.842374 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:33:59.842304    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:34:00.116722 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:00.116694    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"]
Sep 11 18:34:00.118898 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:34:00.118870    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31321600_870d_4814_ab30_121514d58e5d.slice/crio-e13bf0c719e36e11395dcc92fdbd9417852921dd2bb018a374cce759cfe5e481 WatchSource:0}: Error finding container e13bf0c719e36e11395dcc92fdbd9417852921dd2bb018a374cce759cfe5e481: Status 404 returned error can't find the container with id e13bf0c719e36e11395dcc92fdbd9417852921dd2bb018a374cce759cfe5e481
Sep 11 18:34:00.154678 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:00.154656    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:34:00.248971 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:00.248939    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh" event={"ID":"31321600-870d-4814-ab30-121514d58e5d","Type":"ContainerStarted","Data":"b08ea24ebe7be2de0c4fe36a0f0bdcc3d7a4da3a4c8e46961f1d87f9778362b8"}
Sep 11 18:34:00.249118 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:00.248982    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh" event={"ID":"31321600-870d-4814-ab30-121514d58e5d","Type":"ContainerStarted","Data":"e13bf0c719e36e11395dcc92fdbd9417852921dd2bb018a374cce759cfe5e481"}
Sep 11 18:34:00.268146 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:00.268094    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh" podStartSLOduration=1.268079605 podStartE2EDuration="1.268079605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:00.267854885 +0000 UTC m=+1117.008981731" watchObservedRunningTime="2026-09-11 18:34:00.268079605 +0000 UTC m=+1117.009206464"
Sep 11 18:34:09.842905 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:09.842875    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:34:09.843303 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:09.842921    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:34:09.855737 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:09.855712    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:34:10.320713 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:10.320676    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:34:14.023057 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:14.023018    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"]
Sep 11 18:34:14.030244 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:14.030214    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"
Sep 11 18:34:14.033380 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:14.033349    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-18a45f6a\""
Sep 11 18:34:14.040683 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:14.040662    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"]
Sep 11 18:34:14.160061 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:14.160031    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103bde3b-bd85-4e5c-872d-8dbc3b61ef50-plugins-config-volume\") pod \"payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx\" (UID: \"103bde3b-bd85-4e5c-872d-8dbc3b61ef50\") " pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"
Sep 11 18:34:14.160211 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:14.160157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwfqw\" (UniqueName: \"kubernetes.io/projected/103bde3b-bd85-4e5c-872d-8dbc3b61ef50-kube-api-access-nwfqw\") pod \"payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx\" (UID: \"103bde3b-bd85-4e5c-872d-8dbc3b61ef50\") " pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"
Sep 11 18:34:14.260902 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:14.260865    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwfqw\" (UniqueName: \"kubernetes.io/projected/103bde3b-bd85-4e5c-872d-8dbc3b61ef50-kube-api-access-nwfqw\") pod \"payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx\" (UID: \"103bde3b-bd85-4e5c-872d-8dbc3b61ef50\") " pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"
Sep 11 18:34:14.261056 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:14.260959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103bde3b-bd85-4e5c-872d-8dbc3b61ef50-plugins-config-volume\") pod \"payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx\" (UID: \"103bde3b-bd85-4e5c-872d-8dbc3b61ef50\") " pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"
Sep 11 18:34:14.261647 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:14.261630    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103bde3b-bd85-4e5c-872d-8dbc3b61ef50-plugins-config-volume\") pod \"payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx\" (UID: \"103bde3b-bd85-4e5c-872d-8dbc3b61ef50\") " pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"
Sep 11 18:34:14.269888 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:14.269866    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwfqw\" (UniqueName: \"kubernetes.io/projected/103bde3b-bd85-4e5c-872d-8dbc3b61ef50-kube-api-access-nwfqw\") pod \"payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx\" (UID: \"103bde3b-bd85-4e5c-872d-8dbc3b61ef50\") " pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"
Sep 11 18:34:14.344903 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:14.344834    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"
Sep 11 18:34:14.624550 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:14.622290    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"]
Sep 11 18:34:14.626716 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:34:14.626690    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103bde3b_bd85_4e5c_872d_8dbc3b61ef50.slice/crio-95de26174c4c6bf19a2cb8a856eb7369a23ba5c519ab2a64f07ccbd4f6ca0764 WatchSource:0}: Error finding container 95de26174c4c6bf19a2cb8a856eb7369a23ba5c519ab2a64f07ccbd4f6ca0764: Status 404 returned error can't find the container with id 95de26174c4c6bf19a2cb8a856eb7369a23ba5c519ab2a64f07ccbd4f6ca0764
Sep 11 18:34:15.337767 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:15.337727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx" event={"ID":"103bde3b-bd85-4e5c-872d-8dbc3b61ef50","Type":"ContainerStarted","Data":"4ac4ec93e6585fa02f51050157fc268b3cb7b126dcb23fc1a2be63352c894d7f"}
Sep 11 18:34:15.338249 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:15.337790    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx" event={"ID":"103bde3b-bd85-4e5c-872d-8dbc3b61ef50","Type":"ContainerStarted","Data":"95de26174c4c6bf19a2cb8a856eb7369a23ba5c519ab2a64f07ccbd4f6ca0764"}
Sep 11 18:34:15.338249 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:15.337826    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"
Sep 11 18:34:15.354410 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:15.354357    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx" podStartSLOduration=2.3543413060000002 podStartE2EDuration="2.354341306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:15.353436951 +0000 UTC m=+1132.094563796" watchObservedRunningTime="2026-09-11 18:34:15.354341306 +0000 UTC m=+1132.095468149"
Sep 11 18:34:26.345650 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:26.345621    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"
Sep 11 18:34:30.031376 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.031331    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"]
Sep 11 18:34:30.042897 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.042870    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.045582 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.045561    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-100986\"/\"kube-root-ca.crt\""
Sep 11 18:34:30.045768 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.045560    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-100986\"/\"openshift-service-ca.crt\""
Sep 11 18:34:30.046290 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.046266    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-100986\"/\"noten-model-100986-kserve-self-signed-certs\""
Sep 11 18:34:30.046439 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.046288    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-100986\"/\"default-dockercfg-66dfn\""
Sep 11 18:34:30.049159 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.049135    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"]
Sep 11 18:34:30.066154 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.066133    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:34:30.210948 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.210910    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26d7be8e-5941-470d-8845-63177a77be66-tls-certs\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.211197 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.210984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-tmp-dir\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.211197 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.211005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-model-cache\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.211197 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.211027    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt6hq\" (UniqueName: \"kubernetes.io/projected/26d7be8e-5941-470d-8845-63177a77be66-kube-api-access-mt6hq\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.211197 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.211086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-home\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.211197 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.211120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-dshm\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.312144 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.312068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26d7be8e-5941-470d-8845-63177a77be66-tls-certs\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.312292 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.312142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-tmp-dir\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.312292 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.312170    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-model-cache\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.312292 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.312204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt6hq\" (UniqueName: \"kubernetes.io/projected/26d7be8e-5941-470d-8845-63177a77be66-kube-api-access-mt6hq\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.312450 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.312338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-home\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.312450 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.312393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-dshm\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.312670 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.312629    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-tmp-dir\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.313275 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.312725    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-model-cache\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.313275 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.313218    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-home\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.315369 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.315345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-dshm\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.315628 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.315612    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26d7be8e-5941-470d-8845-63177a77be66-tls-certs\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.320538 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.320520    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt6hq\" (UniqueName: \"kubernetes.io/projected/26d7be8e-5941-470d-8845-63177a77be66-kube-api-access-mt6hq\") pod \"noten-model-100986-kserve-5bcd877755-qqfrk\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") " pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.353983 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.353959    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:30.626805 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:30.626777    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"]
Sep 11 18:34:30.627983 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:34:30.627959    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26d7be8e_5941_470d_8845_63177a77be66.slice/crio-b5b57fdf99c291343e447f22015c211dd563b5026746b6d2310c815a172f301b WatchSource:0}: Error finding container b5b57fdf99c291343e447f22015c211dd563b5026746b6d2310c815a172f301b: Status 404 returned error can't find the container with id b5b57fdf99c291343e447f22015c211dd563b5026746b6d2310c815a172f301b
Sep 11 18:34:31.429832 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:31.429791    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk" event={"ID":"26d7be8e-5941-470d-8845-63177a77be66","Type":"ContainerStarted","Data":"5a93275951c670e057e6610864d7e3a7c7e13970af770907f9cf64c4bbf93371"}
Sep 11 18:34:31.429832 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:31.429833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk" event={"ID":"26d7be8e-5941-470d-8845-63177a77be66","Type":"ContainerStarted","Data":"b5b57fdf99c291343e447f22015c211dd563b5026746b6d2310c815a172f301b"}
Sep 11 18:34:31.447725 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:31.447682    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk" podStartSLOduration=1.447666854 podStartE2EDuration="1.447666854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:31.446931892 +0000 UTC m=+1148.188058735" watchObservedRunningTime="2026-09-11 18:34:31.447666854 +0000 UTC m=+1148.188793694"
Sep 11 18:34:40.354535 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:40.354501    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:40.354535 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:40.354542    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:40.367574 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:40.367550    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:34:40.498610 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:34:40.498581    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:35:01.438548 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:01.438456    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"]
Sep 11 18:35:03.577431 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:03.577401    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d7509-29819190-jkx86"]
Sep 11 18:35:03.580765 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:03.580738    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7d7509-29819190-jkx86"]
Sep 11 18:35:03.604719 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:03.604692    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"]
Sep 11 18:35:03.604950 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:03.604898    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j" podUID="253be52b-26e8-4852-969d-de83b2166df7" containerName="payload-processing" containerID="cri-o://4fe023e88ce1b55da5ba0f4324b2fb5f1b6d868c05c1572ab7f507b37747657a" gracePeriod=30
Sep 11 18:35:03.615704 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:03.615677    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"]
Sep 11 18:35:03.615920 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:03.615897    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n" podUID="6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f" containerName="payload-pre-processing" containerID="cri-o://6ab32ce74ccd2a440b56fd9d25a98d2de509ec5d837af4170259b4373614518b" gracePeriod=30
Sep 11 18:35:03.769892 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:03.769817    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="928affc0-ec0f-4bbe-8549-39ee5654cbdc" path="/var/lib/kubelet/pods/928affc0-ec0f-4bbe-8549-39ee5654cbdc/volumes"
Sep 11 18:35:03.899306 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:03.899282    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"
Sep 11 18:35:03.902573 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:03.902555    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"
Sep 11 18:35:04.033787 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.033754    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/253be52b-26e8-4852-969d-de83b2166df7-plugins-config-volume\") pod \"253be52b-26e8-4852-969d-de83b2166df7\" (UID: \"253be52b-26e8-4852-969d-de83b2166df7\") "
Sep 11 18:35:04.033952 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.033834    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dzw2\" (UniqueName: \"kubernetes.io/projected/6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f-kube-api-access-7dzw2\") pod \"6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f\" (UID: \"6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f\") "
Sep 11 18:35:04.033994 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.033951    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f-plugins-config-volume\") pod \"6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f\" (UID: \"6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f\") "
Sep 11 18:35:04.034029 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.033990    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhbds\" (UniqueName: \"kubernetes.io/projected/253be52b-26e8-4852-969d-de83b2166df7-kube-api-access-mhbds\") pod \"253be52b-26e8-4852-969d-de83b2166df7\" (UID: \"253be52b-26e8-4852-969d-de83b2166df7\") "
Sep 11 18:35:04.034309 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.034282    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253be52b-26e8-4852-969d-de83b2166df7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "253be52b-26e8-4852-969d-de83b2166df7" (UID: "253be52b-26e8-4852-969d-de83b2166df7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:35:04.034373 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.034324    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f" (UID: "6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:35:04.036149 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.036124    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f-kube-api-access-7dzw2" (OuterVolumeSpecName: "kube-api-access-7dzw2") pod "6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f" (UID: "6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f"). InnerVolumeSpecName "kube-api-access-7dzw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:04.036205 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.036128    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253be52b-26e8-4852-969d-de83b2166df7-kube-api-access-mhbds" (OuterVolumeSpecName: "kube-api-access-mhbds") pod "253be52b-26e8-4852-969d-de83b2166df7" (UID: "253be52b-26e8-4852-969d-de83b2166df7"). InnerVolumeSpecName "kube-api-access-mhbds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:04.135024 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.135000    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:35:04.135024 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.135021    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhbds\" (UniqueName: \"kubernetes.io/projected/253be52b-26e8-4852-969d-de83b2166df7-kube-api-access-mhbds\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:35:04.135180 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.135032    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/253be52b-26e8-4852-969d-de83b2166df7-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:35:04.135180 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.135040    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dzw2\" (UniqueName: \"kubernetes.io/projected/6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f-kube-api-access-7dzw2\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:35:04.620882 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.620797    2581 generic.go:356] "Generic (PLEG): container finished" podID="253be52b-26e8-4852-969d-de83b2166df7" containerID="4fe023e88ce1b55da5ba0f4324b2fb5f1b6d868c05c1572ab7f507b37747657a" exitCode=0
Sep 11 18:35:04.620882 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.620856    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"
Sep 11 18:35:04.621373 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.620880    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j" event={"ID":"253be52b-26e8-4852-969d-de83b2166df7","Type":"ContainerDied","Data":"4fe023e88ce1b55da5ba0f4324b2fb5f1b6d868c05c1572ab7f507b37747657a"}
Sep 11 18:35:04.621373 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.620993    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j" event={"ID":"253be52b-26e8-4852-969d-de83b2166df7","Type":"ContainerDied","Data":"94dc87e93fe50e8b5306ca2f3eeb8e8aad338d531737cbd218f05152f615d3ff"}
Sep 11 18:35:04.621373 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.621029    2581 scope.go:117] "RemoveContainer" containerID="4fe023e88ce1b55da5ba0f4324b2fb5f1b6d868c05c1572ab7f507b37747657a"
Sep 11 18:35:04.622462 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.622439    2581 generic.go:356] "Generic (PLEG): container finished" podID="6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f" containerID="6ab32ce74ccd2a440b56fd9d25a98d2de509ec5d837af4170259b4373614518b" exitCode=0
Sep 11 18:35:04.622563 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.622521    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n" event={"ID":"6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f","Type":"ContainerDied","Data":"6ab32ce74ccd2a440b56fd9d25a98d2de509ec5d837af4170259b4373614518b"}
Sep 11 18:35:04.622563 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.622529    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"
Sep 11 18:35:04.622834 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.622565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n" event={"ID":"6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f","Type":"ContainerDied","Data":"3963c5915f60bd657f4684f0d01bd47a4daf0b33162a73fd1fdeb1659af9af69"}
Sep 11 18:35:04.639294 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.639278    2581 scope.go:117] "RemoveContainer" containerID="4fe023e88ce1b55da5ba0f4324b2fb5f1b6d868c05c1572ab7f507b37747657a"
Sep 11 18:35:04.639588 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:35:04.639555    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fe023e88ce1b55da5ba0f4324b2fb5f1b6d868c05c1572ab7f507b37747657a\": container with ID starting with 4fe023e88ce1b55da5ba0f4324b2fb5f1b6d868c05c1572ab7f507b37747657a not found: ID does not exist" containerID="4fe023e88ce1b55da5ba0f4324b2fb5f1b6d868c05c1572ab7f507b37747657a"
Sep 11 18:35:04.639689 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.639607    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe023e88ce1b55da5ba0f4324b2fb5f1b6d868c05c1572ab7f507b37747657a"} err="failed to get container status \"4fe023e88ce1b55da5ba0f4324b2fb5f1b6d868c05c1572ab7f507b37747657a\": rpc error: code = NotFound desc = could not find container \"4fe023e88ce1b55da5ba0f4324b2fb5f1b6d868c05c1572ab7f507b37747657a\": container with ID starting with 4fe023e88ce1b55da5ba0f4324b2fb5f1b6d868c05c1572ab7f507b37747657a not found: ID does not exist"
Sep 11 18:35:04.639689 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.639631    2581 scope.go:117] "RemoveContainer" containerID="6ab32ce74ccd2a440b56fd9d25a98d2de509ec5d837af4170259b4373614518b"
Sep 11 18:35:04.648046 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.648031    2581 scope.go:117] "RemoveContainer" containerID="6ab32ce74ccd2a440b56fd9d25a98d2de509ec5d837af4170259b4373614518b"
Sep 11 18:35:04.648330 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:35:04.648305    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab32ce74ccd2a440b56fd9d25a98d2de509ec5d837af4170259b4373614518b\": container with ID starting with 6ab32ce74ccd2a440b56fd9d25a98d2de509ec5d837af4170259b4373614518b not found: ID does not exist" containerID="6ab32ce74ccd2a440b56fd9d25a98d2de509ec5d837af4170259b4373614518b"
Sep 11 18:35:04.648396 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.648327    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab32ce74ccd2a440b56fd9d25a98d2de509ec5d837af4170259b4373614518b"} err="failed to get container status \"6ab32ce74ccd2a440b56fd9d25a98d2de509ec5d837af4170259b4373614518b\": rpc error: code = NotFound desc = could not find container \"6ab32ce74ccd2a440b56fd9d25a98d2de509ec5d837af4170259b4373614518b\": container with ID starting with 6ab32ce74ccd2a440b56fd9d25a98d2de509ec5d837af4170259b4373614518b not found: ID does not exist"
Sep 11 18:35:04.652208 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.652188    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"]
Sep 11 18:35:04.659279 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.659256    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d7509-57c59b5677-w6c6j"]
Sep 11 18:35:04.671953 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.671930    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"]
Sep 11 18:35:04.678521 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:04.678500    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d7509-7767788f78-h894n"]
Sep 11 18:35:05.764008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:05.763977    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="253be52b-26e8-4852-969d-de83b2166df7" path="/var/lib/kubelet/pods/253be52b-26e8-4852-969d-de83b2166df7/volumes"
Sep 11 18:35:05.764378 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:05.764315    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f" path="/var/lib/kubelet/pods/6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f/volumes"
Sep 11 18:35:06.260093 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:06.260048    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:35:16.458287 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.458235    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk" podUID="26d7be8e-5941-470d-8845-63177a77be66" containerName="main" containerID="cri-o://5a93275951c670e057e6610864d7e3a7c7e13970af770907f9cf64c4bbf93371" gracePeriod=45
Sep 11 18:35:16.692461 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.692433    2581 generic.go:356] "Generic (PLEG): container finished" podID="26d7be8e-5941-470d-8845-63177a77be66" containerID="5a93275951c670e057e6610864d7e3a7c7e13970af770907f9cf64c4bbf93371" exitCode=0
Sep 11 18:35:16.692651 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.692490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk" event={"ID":"26d7be8e-5941-470d-8845-63177a77be66","Type":"ContainerDied","Data":"5a93275951c670e057e6610864d7e3a7c7e13970af770907f9cf64c4bbf93371"}
Sep 11 18:35:16.727840 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.727818    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:35:16.867705 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.867666    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26d7be8e-5941-470d-8845-63177a77be66-tls-certs\") pod \"26d7be8e-5941-470d-8845-63177a77be66\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") "
Sep 11 18:35:16.867705 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.867708    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-dshm\") pod \"26d7be8e-5941-470d-8845-63177a77be66\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") "
Sep 11 18:35:16.867897 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.867779    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-model-cache\") pod \"26d7be8e-5941-470d-8845-63177a77be66\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") "
Sep 11 18:35:16.867897 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.867812    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt6hq\" (UniqueName: \"kubernetes.io/projected/26d7be8e-5941-470d-8845-63177a77be66-kube-api-access-mt6hq\") pod \"26d7be8e-5941-470d-8845-63177a77be66\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") "
Sep 11 18:35:16.867987 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.867907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-tmp-dir\") pod \"26d7be8e-5941-470d-8845-63177a77be66\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") "
Sep 11 18:35:16.867987 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.867933    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-home\") pod \"26d7be8e-5941-470d-8845-63177a77be66\" (UID: \"26d7be8e-5941-470d-8845-63177a77be66\") "
Sep 11 18:35:16.868091 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.868017    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-model-cache" (OuterVolumeSpecName: "model-cache") pod "26d7be8e-5941-470d-8845-63177a77be66" (UID: "26d7be8e-5941-470d-8845-63177a77be66"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:16.868229 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.868201    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "26d7be8e-5941-470d-8845-63177a77be66" (UID: "26d7be8e-5941-470d-8845-63177a77be66"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:16.868229 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.868215    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-home" (OuterVolumeSpecName: "home") pod "26d7be8e-5941-470d-8845-63177a77be66" (UID: "26d7be8e-5941-470d-8845-63177a77be66"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:16.868583 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.868562    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:35:16.868687 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.868612    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:35:16.868687 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.868627    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:35:16.870238 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.870210    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-dshm" (OuterVolumeSpecName: "dshm") pod "26d7be8e-5941-470d-8845-63177a77be66" (UID: "26d7be8e-5941-470d-8845-63177a77be66"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:35:16.870238 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.870215    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26d7be8e-5941-470d-8845-63177a77be66-kube-api-access-mt6hq" (OuterVolumeSpecName: "kube-api-access-mt6hq") pod "26d7be8e-5941-470d-8845-63177a77be66" (UID: "26d7be8e-5941-470d-8845-63177a77be66"). InnerVolumeSpecName "kube-api-access-mt6hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:16.870371 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.870339    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26d7be8e-5941-470d-8845-63177a77be66-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "26d7be8e-5941-470d-8845-63177a77be66" (UID: "26d7be8e-5941-470d-8845-63177a77be66"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:35:16.969972 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.969913    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt6hq\" (UniqueName: \"kubernetes.io/projected/26d7be8e-5941-470d-8845-63177a77be66-kube-api-access-mt6hq\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:35:16.969972 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.969935    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26d7be8e-5941-470d-8845-63177a77be66-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:35:16.969972 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:16.969944    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/26d7be8e-5941-470d-8845-63177a77be66-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:35:17.705372 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:17.705337    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"
Sep 11 18:35:17.705372 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:17.705349    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk" event={"ID":"26d7be8e-5941-470d-8845-63177a77be66","Type":"ContainerDied","Data":"b5b57fdf99c291343e447f22015c211dd563b5026746b6d2310c815a172f301b"}
Sep 11 18:35:17.705902 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:17.705401    2581 scope.go:117] "RemoveContainer" containerID="5a93275951c670e057e6610864d7e3a7c7e13970af770907f9cf64c4bbf93371"
Sep 11 18:35:17.729554 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:17.729472    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"]
Sep 11 18:35:17.731796 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:17.731773    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-100986/noten-model-100986-kserve-5bcd877755-qqfrk"]
Sep 11 18:35:17.764254 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:17.764232    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26d7be8e-5941-470d-8845-63177a77be66" path="/var/lib/kubelet/pods/26d7be8e-5941-470d-8845-63177a77be66/volumes"
Sep 11 18:35:21.889327 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:21.889286    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1de19d-29819190-r24gk"]
Sep 11 18:35:21.891358 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:21.891338    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-1de19d-29819190-r24gk"]
Sep 11 18:35:21.919042 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:21.919006    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"]
Sep 11 18:35:21.919255 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:21.919228    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw" podUID="3126d145-7b41-4488-8473-2d83263a3a52" containerName="payload-processing" containerID="cri-o://64b5beb4e7dd035e9a136ad0475b69f84365e168973332912276af258dd331a5" gracePeriod=30
Sep 11 18:35:22.173402 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:22.173381    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"
Sep 11 18:35:22.211793 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:22.211766    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz9tv\" (UniqueName: \"kubernetes.io/projected/3126d145-7b41-4488-8473-2d83263a3a52-kube-api-access-pz9tv\") pod \"3126d145-7b41-4488-8473-2d83263a3a52\" (UID: \"3126d145-7b41-4488-8473-2d83263a3a52\") "
Sep 11 18:35:22.211934 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:22.211831    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3126d145-7b41-4488-8473-2d83263a3a52-plugins-config-volume\") pod \"3126d145-7b41-4488-8473-2d83263a3a52\" (UID: \"3126d145-7b41-4488-8473-2d83263a3a52\") "
Sep 11 18:35:22.212252 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:22.212227    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3126d145-7b41-4488-8473-2d83263a3a52-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3126d145-7b41-4488-8473-2d83263a3a52" (UID: "3126d145-7b41-4488-8473-2d83263a3a52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:35:22.214177 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:22.214134    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3126d145-7b41-4488-8473-2d83263a3a52-kube-api-access-pz9tv" (OuterVolumeSpecName: "kube-api-access-pz9tv") pod "3126d145-7b41-4488-8473-2d83263a3a52" (UID: "3126d145-7b41-4488-8473-2d83263a3a52"). InnerVolumeSpecName "kube-api-access-pz9tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:22.313074 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:22.313048    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz9tv\" (UniqueName: \"kubernetes.io/projected/3126d145-7b41-4488-8473-2d83263a3a52-kube-api-access-pz9tv\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:35:22.313074 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:22.313073    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3126d145-7b41-4488-8473-2d83263a3a52-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:35:22.733990 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:22.733951    2581 generic.go:356] "Generic (PLEG): container finished" podID="3126d145-7b41-4488-8473-2d83263a3a52" containerID="64b5beb4e7dd035e9a136ad0475b69f84365e168973332912276af258dd331a5" exitCode=0
Sep 11 18:35:22.734165 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:22.733995    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"
Sep 11 18:35:22.734165 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:22.734029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw" event={"ID":"3126d145-7b41-4488-8473-2d83263a3a52","Type":"ContainerDied","Data":"64b5beb4e7dd035e9a136ad0475b69f84365e168973332912276af258dd331a5"}
Sep 11 18:35:22.734165 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:22.734082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw" event={"ID":"3126d145-7b41-4488-8473-2d83263a3a52","Type":"ContainerDied","Data":"192d242ac5c7948bde56f3d29cf3b5cf55f967c9fd93b63e3a6b347c1dbd256a"}
Sep 11 18:35:22.734165 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:22.734108    2581 scope.go:117] "RemoveContainer" containerID="64b5beb4e7dd035e9a136ad0475b69f84365e168973332912276af258dd331a5"
Sep 11 18:35:22.743412 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:22.743396    2581 scope.go:117] "RemoveContainer" containerID="64b5beb4e7dd035e9a136ad0475b69f84365e168973332912276af258dd331a5"
Sep 11 18:35:22.743711 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:35:22.743693    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b5beb4e7dd035e9a136ad0475b69f84365e168973332912276af258dd331a5\": container with ID starting with 64b5beb4e7dd035e9a136ad0475b69f84365e168973332912276af258dd331a5 not found: ID does not exist" containerID="64b5beb4e7dd035e9a136ad0475b69f84365e168973332912276af258dd331a5"
Sep 11 18:35:22.743809 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:22.743720    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b5beb4e7dd035e9a136ad0475b69f84365e168973332912276af258dd331a5"} err="failed to get container status \"64b5beb4e7dd035e9a136ad0475b69f84365e168973332912276af258dd331a5\": rpc error: code = NotFound desc = could not find container \"64b5beb4e7dd035e9a136ad0475b69f84365e168973332912276af258dd331a5\": container with ID starting with 64b5beb4e7dd035e9a136ad0475b69f84365e168973332912276af258dd331a5 not found: ID does not exist"
Sep 11 18:35:22.759995 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:22.759971    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"]
Sep 11 18:35:22.767527 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:22.767500    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1de19d-689d49ff7f-l56tw"]
Sep 11 18:35:23.764282 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:23.764247    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3126d145-7b41-4488-8473-2d83263a3a52" path="/var/lib/kubelet/pods/3126d145-7b41-4488-8473-2d83263a3a52/volumes"
Sep 11 18:35:23.764632 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:23.764563    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7430ddfe-7165-4fb9-95fd-1066e668940a" path="/var/lib/kubelet/pods/7430ddfe-7165-4fb9-95fd-1066e668940a/volumes"
Sep 11 18:35:35.443415 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:35.443382    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:35:37.986611 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:37.986574    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"]
Sep 11 18:35:37.987051 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:37.987036    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="253be52b-26e8-4852-969d-de83b2166df7" containerName="payload-processing"
Sep 11 18:35:37.987051 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:37.987052    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="253be52b-26e8-4852-969d-de83b2166df7" containerName="payload-processing"
Sep 11 18:35:37.987127 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:37.987061    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f" containerName="payload-pre-processing"
Sep 11 18:35:37.987127 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:37.987067    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f" containerName="payload-pre-processing"
Sep 11 18:35:37.987127 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:37.987090    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26d7be8e-5941-470d-8845-63177a77be66" containerName="main"
Sep 11 18:35:37.987127 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:37.987096    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="26d7be8e-5941-470d-8845-63177a77be66" containerName="main"
Sep 11 18:35:37.987127 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:37.987108    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3126d145-7b41-4488-8473-2d83263a3a52" containerName="payload-processing"
Sep 11 18:35:37.987127 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:37.987114    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3126d145-7b41-4488-8473-2d83263a3a52" containerName="payload-processing"
Sep 11 18:35:37.987307 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:37.987185    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f8fcdbc-67fa-4ef5-82f6-f3a60666fc7f" containerName="payload-pre-processing"
Sep 11 18:35:37.987307 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:37.987195    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="253be52b-26e8-4852-969d-de83b2166df7" containerName="payload-processing"
Sep 11 18:35:37.987307 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:37.987201    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="26d7be8e-5941-470d-8845-63177a77be66" containerName="main"
Sep 11 18:35:37.987307 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:37.987208    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3126d145-7b41-4488-8473-2d83263a3a52" containerName="payload-processing"
Sep 11 18:35:37.991736 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:37.991720    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"
Sep 11 18:35:37.995606 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:37.995587    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-7ca6df\""
Sep 11 18:35:38.000538 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:38.000509    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"]
Sep 11 18:35:38.062103 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:38.062073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svsst\" (UniqueName: \"kubernetes.io/projected/6e45883f-542d-4cdb-83bb-bf5a063ff3d1-kube-api-access-svsst\") pod \"payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6\" (UID: \"6e45883f-542d-4cdb-83bb-bf5a063ff3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"
Sep 11 18:35:38.062279 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:38.062202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e45883f-542d-4cdb-83bb-bf5a063ff3d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6\" (UID: \"6e45883f-542d-4cdb-83bb-bf5a063ff3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"
Sep 11 18:35:38.163261 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:38.163222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e45883f-542d-4cdb-83bb-bf5a063ff3d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6\" (UID: \"6e45883f-542d-4cdb-83bb-bf5a063ff3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"
Sep 11 18:35:38.163453 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:38.163319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svsst\" (UniqueName: \"kubernetes.io/projected/6e45883f-542d-4cdb-83bb-bf5a063ff3d1-kube-api-access-svsst\") pod \"payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6\" (UID: \"6e45883f-542d-4cdb-83bb-bf5a063ff3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"
Sep 11 18:35:38.164032 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:38.164012    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e45883f-542d-4cdb-83bb-bf5a063ff3d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6\" (UID: \"6e45883f-542d-4cdb-83bb-bf5a063ff3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"
Sep 11 18:35:38.175988 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:38.175962    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svsst\" (UniqueName: \"kubernetes.io/projected/6e45883f-542d-4cdb-83bb-bf5a063ff3d1-kube-api-access-svsst\") pod \"payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6\" (UID: \"6e45883f-542d-4cdb-83bb-bf5a063ff3d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"
Sep 11 18:35:38.302200 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:38.302121    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"
Sep 11 18:35:38.573839 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:38.573770    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"]
Sep 11 18:35:38.577105 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:35:38.577070    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e45883f_542d_4cdb_83bb_bf5a063ff3d1.slice/crio-8440bfcddc1b818d1b549b32bb41e6b25b05d9be93b07a1fe7612d20e053d7dc WatchSource:0}: Error finding container 8440bfcddc1b818d1b549b32bb41e6b25b05d9be93b07a1fe7612d20e053d7dc: Status 404 returned error can't find the container with id 8440bfcddc1b818d1b549b32bb41e6b25b05d9be93b07a1fe7612d20e053d7dc
Sep 11 18:35:38.821320 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:38.821283    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6" event={"ID":"6e45883f-542d-4cdb-83bb-bf5a063ff3d1","Type":"ContainerStarted","Data":"7af88e4495da4252afad9345e162187fa98204e1d735162be184f5d089e61a84"}
Sep 11 18:35:38.821511 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:38.821343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6" event={"ID":"6e45883f-542d-4cdb-83bb-bf5a063ff3d1","Type":"ContainerStarted","Data":"8440bfcddc1b818d1b549b32bb41e6b25b05d9be93b07a1fe7612d20e053d7dc"}
Sep 11 18:35:38.821511 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:38.821414    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"
Sep 11 18:35:38.837913 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:38.837822    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6" podStartSLOduration=1.8378081960000001 podStartE2EDuration="1.837808196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:38.836174957 +0000 UTC m=+1215.577301802" watchObservedRunningTime="2026-09-11 18:35:38.837808196 +0000 UTC m=+1215.578935038"
Sep 11 18:35:48.436794 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.436760    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"]
Sep 11 18:35:48.440441 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.440426    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.443551 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.443525    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-18a45f6a\"/\"e2e-lifecycle-model-18a45f6a-kserve-self-signed-certs\""
Sep 11 18:35:48.443682 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.443551    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-18a45f6a\"/\"kube-root-ca.crt\""
Sep 11 18:35:48.443682 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.443529    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-18a45f6a\"/\"openshift-service-ca.crt\""
Sep 11 18:35:48.443682 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.443525    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-18a45f6a\"/\"default-dockercfg-trgs5\""
Sep 11 18:35:48.448597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.448576    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"]
Sep 11 18:35:48.557863 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.557828    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9mr6\" (UniqueName: \"kubernetes.io/projected/d70fc766-0e61-4932-b7a5-245b3511111b-kube-api-access-g9mr6\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.558055 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.557874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-model-cache\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.558055 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.557905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-tmp-dir\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.558055 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.557973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-home\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.558055 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.558014    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-dshm\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.558055 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.558042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d70fc766-0e61-4932-b7a5-245b3511111b-tls-certs\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.659545 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.659513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-model-cache\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.659733 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.659579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-tmp-dir\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.659733 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.659612    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-home\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.659733 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.659650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-dshm\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.659733 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.659689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d70fc766-0e61-4932-b7a5-245b3511111b-tls-certs\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.659961 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.659794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9mr6\" (UniqueName: \"kubernetes.io/projected/d70fc766-0e61-4932-b7a5-245b3511111b-kube-api-access-g9mr6\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.659961 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.659908    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-model-cache\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.660094 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.660072    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-home\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.660209 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.660187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-tmp-dir\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.662088 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.662068    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-dshm\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.662342 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.662326    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d70fc766-0e61-4932-b7a5-245b3511111b-tls-certs\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.668559 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.668537    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9mr6\" (UniqueName: \"kubernetes.io/projected/d70fc766-0e61-4932-b7a5-245b3511111b-kube-api-access-g9mr6\") pod \"e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") " pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:48.751041 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:48.750977    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:49.023894 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:49.023864    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"]
Sep 11 18:35:49.026355 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:35:49.026327    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd70fc766_0e61_4932_b7a5_245b3511111b.slice/crio-3eaaa0f1a661c4a8c8a0de4eafc3b638d36e2805aaec5d3cce4998e35da8bccf WatchSource:0}: Error finding container 3eaaa0f1a661c4a8c8a0de4eafc3b638d36e2805aaec5d3cce4998e35da8bccf: Status 404 returned error can't find the container with id 3eaaa0f1a661c4a8c8a0de4eafc3b638d36e2805aaec5d3cce4998e35da8bccf
Sep 11 18:35:49.831506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:49.831460    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"
Sep 11 18:35:49.883138 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:49.883106    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc" event={"ID":"d70fc766-0e61-4932-b7a5-245b3511111b","Type":"ContainerStarted","Data":"55bf2077c96e21e015d95f52346b9e1ea03d5068bcba3fbd98bfe2f9bce5d1d2"}
Sep 11 18:35:49.883292 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:49.883149    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc" event={"ID":"d70fc766-0e61-4932-b7a5-245b3511111b","Type":"ContainerStarted","Data":"3eaaa0f1a661c4a8c8a0de4eafc3b638d36e2805aaec5d3cce4998e35da8bccf"}
Sep 11 18:35:49.905997 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:49.905948    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc" podStartSLOduration=1.905931086 podStartE2EDuration="1.905931086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:49.904228804 +0000 UTC m=+1226.645355648" watchObservedRunningTime="2026-09-11 18:35:49.905931086 +0000 UTC m=+1226.647057929"
Sep 11 18:35:58.752095 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:58.752064    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:58.752523 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:58.752129    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:58.768301 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:58.768276    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:35:58.942846 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:35:58.942818    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:36:03.752404 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:03.752372    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:36:13.693509 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.693464    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"]
Sep 11 18:36:13.697277 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.697261    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"
Sep 11 18:36:13.700086 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.700069    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-711a34-dockercfg-pvbs6\""
Sep 11 18:36:13.700987 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.700973    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-711a34\""
Sep 11 18:36:13.714707 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.714682    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"]
Sep 11 18:36:13.742633 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.742610    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"]
Sep 11 18:36:13.746103 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.746089    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"
Sep 11 18:36:13.757277 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.757256    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"]
Sep 11 18:36:13.797111 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.797086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e455243-6075-4137-ba60-a5b81dae2c3e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl\" (UID: \"2e455243-6075-4137-ba60-a5b81dae2c3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"
Sep 11 18:36:13.797235 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.797123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fea0428-cb51-4a39-b775-e0f8833b3e1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv\" (UID: \"0fea0428-cb51-4a39-b775-e0f8833b3e1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"
Sep 11 18:36:13.797235 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.797187    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wngt4\" (UniqueName: \"kubernetes.io/projected/0fea0428-cb51-4a39-b775-e0f8833b3e1b-kube-api-access-wngt4\") pod \"payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv\" (UID: \"0fea0428-cb51-4a39-b775-e0f8833b3e1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"
Sep 11 18:36:13.797235 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.797203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bfwz\" (UniqueName: \"kubernetes.io/projected/2e455243-6075-4137-ba60-a5b81dae2c3e-kube-api-access-5bfwz\") pod \"payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl\" (UID: \"2e455243-6075-4137-ba60-a5b81dae2c3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"
Sep 11 18:36:13.898282 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.898250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e455243-6075-4137-ba60-a5b81dae2c3e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl\" (UID: \"2e455243-6075-4137-ba60-a5b81dae2c3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"
Sep 11 18:36:13.898430 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.898299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fea0428-cb51-4a39-b775-e0f8833b3e1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv\" (UID: \"0fea0428-cb51-4a39-b775-e0f8833b3e1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"
Sep 11 18:36:13.898430 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.898362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wngt4\" (UniqueName: \"kubernetes.io/projected/0fea0428-cb51-4a39-b775-e0f8833b3e1b-kube-api-access-wngt4\") pod \"payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv\" (UID: \"0fea0428-cb51-4a39-b775-e0f8833b3e1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"
Sep 11 18:36:13.898430 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.898387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bfwz\" (UniqueName: \"kubernetes.io/projected/2e455243-6075-4137-ba60-a5b81dae2c3e-kube-api-access-5bfwz\") pod \"payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl\" (UID: \"2e455243-6075-4137-ba60-a5b81dae2c3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"
Sep 11 18:36:13.898961 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.898943    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e455243-6075-4137-ba60-a5b81dae2c3e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl\" (UID: \"2e455243-6075-4137-ba60-a5b81dae2c3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"
Sep 11 18:36:13.899084 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.899066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fea0428-cb51-4a39-b775-e0f8833b3e1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv\" (UID: \"0fea0428-cb51-4a39-b775-e0f8833b3e1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"
Sep 11 18:36:13.908046 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.908025    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wngt4\" (UniqueName: \"kubernetes.io/projected/0fea0428-cb51-4a39-b775-e0f8833b3e1b-kube-api-access-wngt4\") pod \"payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv\" (UID: \"0fea0428-cb51-4a39-b775-e0f8833b3e1b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"
Sep 11 18:36:13.908161 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:13.908085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bfwz\" (UniqueName: \"kubernetes.io/projected/2e455243-6075-4137-ba60-a5b81dae2c3e-kube-api-access-5bfwz\") pod \"payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl\" (UID: \"2e455243-6075-4137-ba60-a5b81dae2c3e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"
Sep 11 18:36:14.007545 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:14.007423    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"
Sep 11 18:36:14.056661 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:14.056631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"
Sep 11 18:36:14.286108 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:14.286084    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"]
Sep 11 18:36:14.287851 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:36:14.287825    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e455243_6075_4137_ba60_a5b81dae2c3e.slice/crio-3a429a6c06e2d326ece2af1db8950611a0168e651a1852fa3e44b005df724382 WatchSource:0}: Error finding container 3a429a6c06e2d326ece2af1db8950611a0168e651a1852fa3e44b005df724382: Status 404 returned error can't find the container with id 3a429a6c06e2d326ece2af1db8950611a0168e651a1852fa3e44b005df724382
Sep 11 18:36:14.339681 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:14.339653    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"]
Sep 11 18:36:14.341534 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:36:14.341506    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fea0428_cb51_4a39_b775_e0f8833b3e1b.slice/crio-f0e83e8bdb12a1b02116826a6647a76a5ece92db271206bea2136f6fc6c7b47b WatchSource:0}: Error finding container f0e83e8bdb12a1b02116826a6647a76a5ece92db271206bea2136f6fc6c7b47b: Status 404 returned error can't find the container with id f0e83e8bdb12a1b02116826a6647a76a5ece92db271206bea2136f6fc6c7b47b
Sep 11 18:36:15.020208 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:15.020115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl" event={"ID":"2e455243-6075-4137-ba60-a5b81dae2c3e","Type":"ContainerStarted","Data":"180884c3d311e815d7ed5c2e59c233f5c287fec36c8af06befd7450122b2fd40"}
Sep 11 18:36:15.020208 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:15.020180    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"
Sep 11 18:36:15.020208 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:15.020206    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl" event={"ID":"2e455243-6075-4137-ba60-a5b81dae2c3e","Type":"ContainerStarted","Data":"3a429a6c06e2d326ece2af1db8950611a0168e651a1852fa3e44b005df724382"}
Sep 11 18:36:15.025682 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:15.025655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv" event={"ID":"0fea0428-cb51-4a39-b775-e0f8833b3e1b","Type":"ContainerStarted","Data":"740fb34f4a21e17c5752e329c929fc516a51ec197a22299849679c314bef8011"}
Sep 11 18:36:15.025826 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:15.025708    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv" event={"ID":"0fea0428-cb51-4a39-b775-e0f8833b3e1b","Type":"ContainerStarted","Data":"f0e83e8bdb12a1b02116826a6647a76a5ece92db271206bea2136f6fc6c7b47b"}
Sep 11 18:36:15.025826 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:15.025736    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"
Sep 11 18:36:15.038525 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:15.038465    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl" podStartSLOduration=2.038450415 podStartE2EDuration="2.038450415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:15.036455263 +0000 UTC m=+1251.777582107" watchObservedRunningTime="2026-09-11 18:36:15.038450415 +0000 UTC m=+1251.779577258"
Sep 11 18:36:15.054839 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:15.054801    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv" podStartSLOduration=2.054788363 podStartE2EDuration="2.054788363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:15.05267753 +0000 UTC m=+1251.793804373" watchObservedRunningTime="2026-09-11 18:36:15.054788363 +0000 UTC m=+1251.795915205"
Sep 11 18:36:23.847487 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:23.847456    2581 scope.go:117] "RemoveContainer" containerID="a3037f1df9bc6cf3b824480cb40c51d24fa2e589ffdd58233d4628b73e735be2"
Sep 11 18:36:23.856158 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:23.856135    2581 scope.go:117] "RemoveContainer" containerID="052084acb90fd9b3e56c80c857a6424bd2167c71e5bac7408028c520fde5b5fa"
Sep 11 18:36:26.032857 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:26.032825    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"
Sep 11 18:36:26.033322 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:26.032993    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"
Sep 11 18:36:26.056716 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:26.056684    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:36:50.946762 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:50.946725    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:36:58.926142 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:36:58.926107    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"]
Sep 11 18:37:13.946506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:13.946445    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb" podUID="38281df6-6cb3-4859-bb54-dfd57447d45b" containerName="main" containerID="cri-o://886b9bf18d47c73b22f3da90dc1c469fc12d4cbe0a85240881e8ef59f98cdbac" gracePeriod=45
Sep 11 18:37:13.977316 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:13.977273    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb" podUID="38281df6-6cb3-4859-bb54-dfd57447d45b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": dial tcp 10.134.0.62:8000: connect: connection refused"
Sep 11 18:37:14.201784 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.201731    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:37:14.344143 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.344111    2581 generic.go:356] "Generic (PLEG): container finished" podID="38281df6-6cb3-4859-bb54-dfd57447d45b" containerID="886b9bf18d47c73b22f3da90dc1c469fc12d4cbe0a85240881e8ef59f98cdbac" exitCode=0
Sep 11 18:37:14.344311 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.344157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb" event={"ID":"38281df6-6cb3-4859-bb54-dfd57447d45b","Type":"ContainerDied","Data":"886b9bf18d47c73b22f3da90dc1c469fc12d4cbe0a85240881e8ef59f98cdbac"}
Sep 11 18:37:14.344311 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.344167    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"
Sep 11 18:37:14.344311 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.344191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb" event={"ID":"38281df6-6cb3-4859-bb54-dfd57447d45b","Type":"ContainerDied","Data":"21e965cb2d0c545af6776ffc0ccf7e4f69c34c6de4fd62c084dfe15e65fe28e3"}
Sep 11 18:37:14.344311 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.344211    2581 scope.go:117] "RemoveContainer" containerID="886b9bf18d47c73b22f3da90dc1c469fc12d4cbe0a85240881e8ef59f98cdbac"
Sep 11 18:37:14.353359 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.353342    2581 scope.go:117] "RemoveContainer" containerID="886b9bf18d47c73b22f3da90dc1c469fc12d4cbe0a85240881e8ef59f98cdbac"
Sep 11 18:37:14.353770 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:37:14.353750    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886b9bf18d47c73b22f3da90dc1c469fc12d4cbe0a85240881e8ef59f98cdbac\": container with ID starting with 886b9bf18d47c73b22f3da90dc1c469fc12d4cbe0a85240881e8ef59f98cdbac not found: ID does not exist" containerID="886b9bf18d47c73b22f3da90dc1c469fc12d4cbe0a85240881e8ef59f98cdbac"
Sep 11 18:37:14.353857 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.353777    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886b9bf18d47c73b22f3da90dc1c469fc12d4cbe0a85240881e8ef59f98cdbac"} err="failed to get container status \"886b9bf18d47c73b22f3da90dc1c469fc12d4cbe0a85240881e8ef59f98cdbac\": rpc error: code = NotFound desc = could not find container \"886b9bf18d47c73b22f3da90dc1c469fc12d4cbe0a85240881e8ef59f98cdbac\": container with ID starting with 886b9bf18d47c73b22f3da90dc1c469fc12d4cbe0a85240881e8ef59f98cdbac not found: ID does not exist"
Sep 11 18:37:14.378132 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.378111    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-home\") pod \"38281df6-6cb3-4859-bb54-dfd57447d45b\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") "
Sep 11 18:37:14.378235 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.378152    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-model-cache\") pod \"38281df6-6cb3-4859-bb54-dfd57447d45b\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") "
Sep 11 18:37:14.378235 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.378211    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9ftm\" (UniqueName: \"kubernetes.io/projected/38281df6-6cb3-4859-bb54-dfd57447d45b-kube-api-access-d9ftm\") pod \"38281df6-6cb3-4859-bb54-dfd57447d45b\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") "
Sep 11 18:37:14.378315 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.378273    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38281df6-6cb3-4859-bb54-dfd57447d45b-tls-certs\") pod \"38281df6-6cb3-4859-bb54-dfd57447d45b\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") "
Sep 11 18:37:14.378390 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.378376    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-tmp-dir\") pod \"38281df6-6cb3-4859-bb54-dfd57447d45b\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") "
Sep 11 18:37:14.378444 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.378364    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-home" (OuterVolumeSpecName: "home") pod "38281df6-6cb3-4859-bb54-dfd57447d45b" (UID: "38281df6-6cb3-4859-bb54-dfd57447d45b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:14.378444 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.378419    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-model-cache" (OuterVolumeSpecName: "model-cache") pod "38281df6-6cb3-4859-bb54-dfd57447d45b" (UID: "38281df6-6cb3-4859-bb54-dfd57447d45b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:14.378580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.378453    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-dshm\") pod \"38281df6-6cb3-4859-bb54-dfd57447d45b\" (UID: \"38281df6-6cb3-4859-bb54-dfd57447d45b\") "
Sep 11 18:37:14.378642 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.378588    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38281df6-6cb3-4859-bb54-dfd57447d45b" (UID: "38281df6-6cb3-4859-bb54-dfd57447d45b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:14.378897 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.378873    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:37:14.378897 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.378902    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:37:14.378897 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.378916    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:37:14.380567 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.380546    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-dshm" (OuterVolumeSpecName: "dshm") pod "38281df6-6cb3-4859-bb54-dfd57447d45b" (UID: "38281df6-6cb3-4859-bb54-dfd57447d45b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:14.380660 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.380594    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38281df6-6cb3-4859-bb54-dfd57447d45b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38281df6-6cb3-4859-bb54-dfd57447d45b" (UID: "38281df6-6cb3-4859-bb54-dfd57447d45b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:37:14.380660 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.380596    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38281df6-6cb3-4859-bb54-dfd57447d45b-kube-api-access-d9ftm" (OuterVolumeSpecName: "kube-api-access-d9ftm") pod "38281df6-6cb3-4859-bb54-dfd57447d45b" (UID: "38281df6-6cb3-4859-bb54-dfd57447d45b"). InnerVolumeSpecName "kube-api-access-d9ftm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:14.479763 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.479694    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38281df6-6cb3-4859-bb54-dfd57447d45b-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:37:14.479763 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.479718    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9ftm\" (UniqueName: \"kubernetes.io/projected/38281df6-6cb3-4859-bb54-dfd57447d45b-kube-api-access-d9ftm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:37:14.479763 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.479729    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38281df6-6cb3-4859-bb54-dfd57447d45b-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:37:14.667268 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.667238    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"]
Sep 11 18:37:14.672325 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:14.672302    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-724327/e2e-conflict-a-498c8da2-kserve-549bc975d-r4fpb"]
Sep 11 18:37:15.764646 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:15.764612    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38281df6-6cb3-4859-bb54-dfd57447d45b" path="/var/lib/kubelet/pods/38281df6-6cb3-4859-bb54-dfd57447d45b/volumes"
Sep 11 18:37:16.409691 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:16.409663    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx"]
Sep 11 18:37:16.410128 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:16.410116    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38281df6-6cb3-4859-bb54-dfd57447d45b" containerName="main"
Sep 11 18:37:16.410174 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:16.410130    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="38281df6-6cb3-4859-bb54-dfd57447d45b" containerName="main"
Sep 11 18:37:16.410212 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:16.410203    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="38281df6-6cb3-4859-bb54-dfd57447d45b" containerName="main"
Sep 11 18:37:16.414277 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:16.414261    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx"
Sep 11 18:37:16.416330 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:16.416312    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:37:16.416405 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:16.416350    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\""
Sep 11 18:37:16.425921 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:16.425896    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx"]
Sep 11 18:37:16.598686 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:16.598656    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23f4c71-7f90-4445-a453-ed09938b420d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx\" (UID: \"e23f4c71-7f90-4445-a453-ed09938b420d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx"
Sep 11 18:37:16.699736 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:16.699654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23f4c71-7f90-4445-a453-ed09938b420d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx\" (UID: \"e23f4c71-7f90-4445-a453-ed09938b420d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx"
Sep 11 18:37:16.700210 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:16.700192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23f4c71-7f90-4445-a453-ed09938b420d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx\" (UID: \"e23f4c71-7f90-4445-a453-ed09938b420d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx"
Sep 11 18:37:16.725521 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:16.725457    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx"
Sep 11 18:37:16.996126 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:16.996105    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx"]
Sep 11 18:37:16.998043 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:37:16.998013    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23f4c71_7f90_4445_a453_ed09938b420d.slice/crio-30f5156f88c531958bbd2ec6f6dab5ad8c00f5142cfdc1ab8442a5ae80940749 WatchSource:0}: Error finding container 30f5156f88c531958bbd2ec6f6dab5ad8c00f5142cfdc1ab8442a5ae80940749: Status 404 returned error can't find the container with id 30f5156f88c531958bbd2ec6f6dab5ad8c00f5142cfdc1ab8442a5ae80940749
Sep 11 18:37:17.362597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:17.362514    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx" event={"ID":"e23f4c71-7f90-4445-a453-ed09938b420d","Type":"ContainerStarted","Data":"004cb877ba59c46363f65d49bdba1d5daf2abfd361f6fc7121830cd2fcaecb87"}
Sep 11 18:37:17.362597 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:17.362563    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx" event={"ID":"e23f4c71-7f90-4445-a453-ed09938b420d","Type":"ContainerStarted","Data":"30f5156f88c531958bbd2ec6f6dab5ad8c00f5142cfdc1ab8442a5ae80940749"}
Sep 11 18:37:17.378822 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:17.378773    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx" podStartSLOduration=1.378758299 podStartE2EDuration="1.378758299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:37:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:37:17.376304825 +0000 UTC m=+1314.117431669" watchObservedRunningTime="2026-09-11 18:37:17.378758299 +0000 UTC m=+1314.119885142"
Sep 11 18:37:18.369844 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:18.369808    2581 generic.go:356] "Generic (PLEG): container finished" podID="e23f4c71-7f90-4445-a453-ed09938b420d" containerID="004cb877ba59c46363f65d49bdba1d5daf2abfd361f6fc7121830cd2fcaecb87" exitCode=0
Sep 11 18:37:18.370199 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:18.369895    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx" event={"ID":"e23f4c71-7f90-4445-a453-ed09938b420d","Type":"ContainerDied","Data":"004cb877ba59c46363f65d49bdba1d5daf2abfd361f6fc7121830cd2fcaecb87"}
Sep 11 18:37:19.509646 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:19.509624    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx"
Sep 11 18:37:19.526348 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:19.526319    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23f4c71-7f90-4445-a453-ed09938b420d-maas-api-service-ca\") pod \"e23f4c71-7f90-4445-a453-ed09938b420d\" (UID: \"e23f4c71-7f90-4445-a453-ed09938b420d\") "
Sep 11 18:37:19.526963 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:19.526940    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23f4c71-7f90-4445-a453-ed09938b420d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e23f4c71-7f90-4445-a453-ed09938b420d" (UID: "e23f4c71-7f90-4445-a453-ed09938b420d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:37:19.627591 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:19.627517    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23f4c71-7f90-4445-a453-ed09938b420d-maas-api-service-ca\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:37:20.383245 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:20.383216    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx" event={"ID":"e23f4c71-7f90-4445-a453-ed09938b420d","Type":"ContainerDied","Data":"30f5156f88c531958bbd2ec6f6dab5ad8c00f5142cfdc1ab8442a5ae80940749"}
Sep 11 18:37:20.383245 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:20.383241    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx"
Sep 11 18:37:20.383245 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:20.383261    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30f5156f88c531958bbd2ec6f6dab5ad8c00f5142cfdc1ab8442a5ae80940749"
Sep 11 18:37:27.844709 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:27.844678    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:37:32.051782 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.051745    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"]
Sep 11 18:37:32.052277 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.052004    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx" podUID="103bde3b-bd85-4e5c-872d-8dbc3b61ef50" containerName="payload-processing" containerID="cri-o://4ac4ec93e6585fa02f51050157fc268b3cb7b126dcb23fc1a2be63352c894d7f" gracePeriod=30
Sep 11 18:37:32.241370 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.241321    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx"]
Sep 11 18:37:32.244272 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.244237    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-18a45f6a-74qhx"]
Sep 11 18:37:32.295148 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.295129    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"
Sep 11 18:37:32.354001 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.353942    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwfqw\" (UniqueName: \"kubernetes.io/projected/103bde3b-bd85-4e5c-872d-8dbc3b61ef50-kube-api-access-nwfqw\") pod \"103bde3b-bd85-4e5c-872d-8dbc3b61ef50\" (UID: \"103bde3b-bd85-4e5c-872d-8dbc3b61ef50\") "
Sep 11 18:37:32.354121 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.354091    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103bde3b-bd85-4e5c-872d-8dbc3b61ef50-plugins-config-volume\") pod \"103bde3b-bd85-4e5c-872d-8dbc3b61ef50\" (UID: \"103bde3b-bd85-4e5c-872d-8dbc3b61ef50\") "
Sep 11 18:37:32.354411 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.354392    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103bde3b-bd85-4e5c-872d-8dbc3b61ef50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "103bde3b-bd85-4e5c-872d-8dbc3b61ef50" (UID: "103bde3b-bd85-4e5c-872d-8dbc3b61ef50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:37:32.354508 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.354471    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/103bde3b-bd85-4e5c-872d-8dbc3b61ef50-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:37:32.356232 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.356205    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103bde3b-bd85-4e5c-872d-8dbc3b61ef50-kube-api-access-nwfqw" (OuterVolumeSpecName: "kube-api-access-nwfqw") pod "103bde3b-bd85-4e5c-872d-8dbc3b61ef50" (UID: "103bde3b-bd85-4e5c-872d-8dbc3b61ef50"). InnerVolumeSpecName "kube-api-access-nwfqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:32.450048 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.450016    2581 generic.go:356] "Generic (PLEG): container finished" podID="103bde3b-bd85-4e5c-872d-8dbc3b61ef50" containerID="4ac4ec93e6585fa02f51050157fc268b3cb7b126dcb23fc1a2be63352c894d7f" exitCode=0
Sep 11 18:37:32.450205 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.450059    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"
Sep 11 18:37:32.450205 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.450101    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx" event={"ID":"103bde3b-bd85-4e5c-872d-8dbc3b61ef50","Type":"ContainerDied","Data":"4ac4ec93e6585fa02f51050157fc268b3cb7b126dcb23fc1a2be63352c894d7f"}
Sep 11 18:37:32.450205 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.450155    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx" event={"ID":"103bde3b-bd85-4e5c-872d-8dbc3b61ef50","Type":"ContainerDied","Data":"95de26174c4c6bf19a2cb8a856eb7369a23ba5c519ab2a64f07ccbd4f6ca0764"}
Sep 11 18:37:32.450205 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.450176    2581 scope.go:117] "RemoveContainer" containerID="4ac4ec93e6585fa02f51050157fc268b3cb7b126dcb23fc1a2be63352c894d7f"
Sep 11 18:37:32.455155 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.455130    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwfqw\" (UniqueName: \"kubernetes.io/projected/103bde3b-bd85-4e5c-872d-8dbc3b61ef50-kube-api-access-nwfqw\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:37:32.459710 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.459690    2581 scope.go:117] "RemoveContainer" containerID="4ac4ec93e6585fa02f51050157fc268b3cb7b126dcb23fc1a2be63352c894d7f"
Sep 11 18:37:32.459985 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:37:32.459966    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac4ec93e6585fa02f51050157fc268b3cb7b126dcb23fc1a2be63352c894d7f\": container with ID starting with 4ac4ec93e6585fa02f51050157fc268b3cb7b126dcb23fc1a2be63352c894d7f not found: ID does not exist" containerID="4ac4ec93e6585fa02f51050157fc268b3cb7b126dcb23fc1a2be63352c894d7f"
Sep 11 18:37:32.460071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.459992    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac4ec93e6585fa02f51050157fc268b3cb7b126dcb23fc1a2be63352c894d7f"} err="failed to get container status \"4ac4ec93e6585fa02f51050157fc268b3cb7b126dcb23fc1a2be63352c894d7f\": rpc error: code = NotFound desc = could not find container \"4ac4ec93e6585fa02f51050157fc268b3cb7b126dcb23fc1a2be63352c894d7f\": container with ID starting with 4ac4ec93e6585fa02f51050157fc268b3cb7b126dcb23fc1a2be63352c894d7f not found: ID does not exist"
Sep 11 18:37:32.479316 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.479291    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"]
Sep 11 18:37:32.484730 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:32.484709    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-18a45f6a-bcb84bb57-559bx"]
Sep 11 18:37:33.764951 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:33.764917    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="103bde3b-bd85-4e5c-872d-8dbc3b61ef50" path="/var/lib/kubelet/pods/103bde3b-bd85-4e5c-872d-8dbc3b61ef50/volumes"
Sep 11 18:37:33.765324 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:33.765239    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e23f4c71-7f90-4445-a453-ed09938b420d" path="/var/lib/kubelet/pods/e23f4c71-7f90-4445-a453-ed09938b420d/volumes"
Sep 11 18:37:40.251813 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:40.251779    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"]
Sep 11 18:37:42.142193 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.142153    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"]
Sep 11 18:37:42.142632 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.142619    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e23f4c71-7f90-4445-a453-ed09938b420d" containerName="revoke-keys"
Sep 11 18:37:42.142632 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.142634    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e23f4c71-7f90-4445-a453-ed09938b420d" containerName="revoke-keys"
Sep 11 18:37:42.142729 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.142648    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103bde3b-bd85-4e5c-872d-8dbc3b61ef50" containerName="payload-processing"
Sep 11 18:37:42.142729 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.142654    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="103bde3b-bd85-4e5c-872d-8dbc3b61ef50" containerName="payload-processing"
Sep 11 18:37:42.142729 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.142725    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="103bde3b-bd85-4e5c-872d-8dbc3b61ef50" containerName="payload-processing"
Sep 11 18:37:42.142833 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.142734    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e23f4c71-7f90-4445-a453-ed09938b420d" containerName="revoke-keys"
Sep 11 18:37:42.147628 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.147613    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.149558 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.149537    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-724327\"/\"distinct-w4-724327-kserve-self-signed-certs\""
Sep 11 18:37:42.153728 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.153707    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"]
Sep 11 18:37:42.247344 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.247304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-tmp-dir\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.247552 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.247355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bb583ad-8821-4669-b229-7e57b8b94add-tls-certs\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.247552 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.247444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-home\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.247552 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.247495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-dshm\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.247552 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.247515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btgrj\" (UniqueName: \"kubernetes.io/projected/2bb583ad-8821-4669-b229-7e57b8b94add-kube-api-access-btgrj\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.247719 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.247558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-model-cache\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.348085 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.348051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-home\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.348237 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.348092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-dshm\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.348237 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.348115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btgrj\" (UniqueName: \"kubernetes.io/projected/2bb583ad-8821-4669-b229-7e57b8b94add-kube-api-access-btgrj\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.348237 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.348153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-model-cache\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.348411 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.348244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-tmp-dir\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.348411 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.348298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bb583ad-8821-4669-b229-7e57b8b94add-tls-certs\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.348589 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.348434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-home\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.348589 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.348556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-model-cache\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.348705 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.348661    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-tmp-dir\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.350557 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.350538    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-dshm\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.350839 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.350821    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bb583ad-8821-4669-b229-7e57b8b94add-tls-certs\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.355842 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.355822    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btgrj\" (UniqueName: \"kubernetes.io/projected/2bb583ad-8821-4669-b229-7e57b8b94add-kube-api-access-btgrj\") pod \"distinct-w4-724327-kserve-85d76d749b-5nxqz\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") " pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.458718 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.458664    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:42.729890 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:42.729868    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"]
Sep 11 18:37:42.731954 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:37:42.731921    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2bb583ad_8821_4669_b229_7e57b8b94add.slice/crio-83144454850ab6fbfe0322f841b3f453a74ff15829b13d3d8abd7520d54b9e9e WatchSource:0}: Error finding container 83144454850ab6fbfe0322f841b3f453a74ff15829b13d3d8abd7520d54b9e9e: Status 404 returned error can't find the container with id 83144454850ab6fbfe0322f841b3f453a74ff15829b13d3d8abd7520d54b9e9e
Sep 11 18:37:43.514462 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:43.514425    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz" event={"ID":"2bb583ad-8821-4669-b229-7e57b8b94add","Type":"ContainerStarted","Data":"51011e755f52ba121d18f7f83ffcfc2bbda47eb3a50f512a5e5b0168d2c26d5f"}
Sep 11 18:37:43.514878 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:43.514493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz" event={"ID":"2bb583ad-8821-4669-b229-7e57b8b94add","Type":"ContainerStarted","Data":"83144454850ab6fbfe0322f841b3f453a74ff15829b13d3d8abd7520d54b9e9e"}
Sep 11 18:37:43.531360 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:43.531308    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz" podStartSLOduration=1.5312959259999999 podStartE2EDuration="1.531295926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:37:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:37:43.529650959 +0000 UTC m=+1340.270777804" watchObservedRunningTime="2026-09-11 18:37:43.531295926 +0000 UTC m=+1340.272422799"
Sep 11 18:37:52.458983 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:52.458904    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:52.458983 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:52.458951    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:52.471622 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:52.471601    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:52.573148 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:52.573126    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:37:55.271966 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.271900    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc" podUID="d70fc766-0e61-4932-b7a5-245b3511111b" containerName="main" containerID="cri-o://55bf2077c96e21e015d95f52346b9e1ea03d5068bcba3fbd98bfe2f9bce5d1d2" gracePeriod=45
Sep 11 18:37:55.531513 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.531447    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:37:55.571670 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.571641    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-home\") pod \"d70fc766-0e61-4932-b7a5-245b3511111b\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") "
Sep 11 18:37:55.571831 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.571730    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-dshm\") pod \"d70fc766-0e61-4932-b7a5-245b3511111b\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") "
Sep 11 18:37:55.571831 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.571825    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-tmp-dir\") pod \"d70fc766-0e61-4932-b7a5-245b3511111b\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") "
Sep 11 18:37:55.571960 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.571850    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d70fc766-0e61-4932-b7a5-245b3511111b-tls-certs\") pod \"d70fc766-0e61-4932-b7a5-245b3511111b\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") "
Sep 11 18:37:55.571960 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.571879    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9mr6\" (UniqueName: \"kubernetes.io/projected/d70fc766-0e61-4932-b7a5-245b3511111b-kube-api-access-g9mr6\") pod \"d70fc766-0e61-4932-b7a5-245b3511111b\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") "
Sep 11 18:37:55.571960 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.571909    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-model-cache\") pod \"d70fc766-0e61-4932-b7a5-245b3511111b\" (UID: \"d70fc766-0e61-4932-b7a5-245b3511111b\") "
Sep 11 18:37:55.572139 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.572113    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d70fc766-0e61-4932-b7a5-245b3511111b" (UID: "d70fc766-0e61-4932-b7a5-245b3511111b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:55.572272 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.572251    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:37:55.572394 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.572367    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d70fc766-0e61-4932-b7a5-245b3511111b" (UID: "d70fc766-0e61-4932-b7a5-245b3511111b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:55.572557 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.572442    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-home" (OuterVolumeSpecName: "home") pod "d70fc766-0e61-4932-b7a5-245b3511111b" (UID: "d70fc766-0e61-4932-b7a5-245b3511111b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:55.574313 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.574282    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-dshm" (OuterVolumeSpecName: "dshm") pod "d70fc766-0e61-4932-b7a5-245b3511111b" (UID: "d70fc766-0e61-4932-b7a5-245b3511111b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:55.574733 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.574709    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d70fc766-0e61-4932-b7a5-245b3511111b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d70fc766-0e61-4932-b7a5-245b3511111b" (UID: "d70fc766-0e61-4932-b7a5-245b3511111b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:37:55.575271 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.575243    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70fc766-0e61-4932-b7a5-245b3511111b-kube-api-access-g9mr6" (OuterVolumeSpecName: "kube-api-access-g9mr6") pod "d70fc766-0e61-4932-b7a5-245b3511111b" (UID: "d70fc766-0e61-4932-b7a5-245b3511111b"). InnerVolumeSpecName "kube-api-access-g9mr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:55.582392 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.582367    2581 generic.go:356] "Generic (PLEG): container finished" podID="d70fc766-0e61-4932-b7a5-245b3511111b" containerID="55bf2077c96e21e015d95f52346b9e1ea03d5068bcba3fbd98bfe2f9bce5d1d2" exitCode=0
Sep 11 18:37:55.582531 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.582415    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"
Sep 11 18:37:55.582531 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.582456    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc" event={"ID":"d70fc766-0e61-4932-b7a5-245b3511111b","Type":"ContainerDied","Data":"55bf2077c96e21e015d95f52346b9e1ea03d5068bcba3fbd98bfe2f9bce5d1d2"}
Sep 11 18:37:55.582657 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.582545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc" event={"ID":"d70fc766-0e61-4932-b7a5-245b3511111b","Type":"ContainerDied","Data":"3eaaa0f1a661c4a8c8a0de4eafc3b638d36e2805aaec5d3cce4998e35da8bccf"}
Sep 11 18:37:55.582657 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.582569    2581 scope.go:117] "RemoveContainer" containerID="55bf2077c96e21e015d95f52346b9e1ea03d5068bcba3fbd98bfe2f9bce5d1d2"
Sep 11 18:37:55.598098 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.598043    2581 scope.go:117] "RemoveContainer" containerID="55bf2077c96e21e015d95f52346b9e1ea03d5068bcba3fbd98bfe2f9bce5d1d2"
Sep 11 18:37:55.598412 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:37:55.598389    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55bf2077c96e21e015d95f52346b9e1ea03d5068bcba3fbd98bfe2f9bce5d1d2\": container with ID starting with 55bf2077c96e21e015d95f52346b9e1ea03d5068bcba3fbd98bfe2f9bce5d1d2 not found: ID does not exist" containerID="55bf2077c96e21e015d95f52346b9e1ea03d5068bcba3fbd98bfe2f9bce5d1d2"
Sep 11 18:37:55.598575 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.598419    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55bf2077c96e21e015d95f52346b9e1ea03d5068bcba3fbd98bfe2f9bce5d1d2"} err="failed to get container status \"55bf2077c96e21e015d95f52346b9e1ea03d5068bcba3fbd98bfe2f9bce5d1d2\": rpc error: code = NotFound desc = could not find container \"55bf2077c96e21e015d95f52346b9e1ea03d5068bcba3fbd98bfe2f9bce5d1d2\": container with ID starting with 55bf2077c96e21e015d95f52346b9e1ea03d5068bcba3fbd98bfe2f9bce5d1d2 not found: ID does not exist"
Sep 11 18:37:55.610209 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.610188    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"]
Sep 11 18:37:55.614158 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.614135    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-18a45f6a/e2e-lifecycle-model-18a45f6a-kserve-5c4f5894bc-mt5qc"]
Sep 11 18:37:55.673583 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.673558    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d70fc766-0e61-4932-b7a5-245b3511111b-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:37:55.673583 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.673583    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9mr6\" (UniqueName: \"kubernetes.io/projected/d70fc766-0e61-4932-b7a5-245b3511111b-kube-api-access-g9mr6\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:37:55.673715 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.673596    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:37:55.673715 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.673605    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:37:55.673715 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.673613    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d70fc766-0e61-4932-b7a5-245b3511111b-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:37:55.764678 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:55.764647    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d70fc766-0e61-4932-b7a5-245b3511111b" path="/var/lib/kubelet/pods/d70fc766-0e61-4932-b7a5-245b3511111b/volumes"
Sep 11 18:37:59.147089 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.147057    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"]
Sep 11 18:37:59.147719 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.147699    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d70fc766-0e61-4932-b7a5-245b3511111b" containerName="main"
Sep 11 18:37:59.147822 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.147722    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d70fc766-0e61-4932-b7a5-245b3511111b" containerName="main"
Sep 11 18:37:59.147885 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.147827    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d70fc766-0e61-4932-b7a5-245b3511111b" containerName="main"
Sep 11 18:37:59.152539 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.152511    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.155429 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.155407    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-711a34\"/\"default-dockercfg-f7z6v\""
Sep 11 18:37:59.155429 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.155418    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-711a34\"/\"auth-test-model-711a34-kserve-self-signed-certs\""
Sep 11 18:37:59.155631 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.155422    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-711a34\"/\"openshift-service-ca.crt\""
Sep 11 18:37:59.155631 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.155415    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-711a34\"/\"kube-root-ca.crt\""
Sep 11 18:37:59.159763 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.159741    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"]
Sep 11 18:37:59.209319 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.209286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k8b8\" (UniqueName: \"kubernetes.io/projected/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-kube-api-access-8k8b8\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.209465 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.209347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-tmp-dir\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.209465 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.209382    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-dshm\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.209465 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.209431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-home\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.209465 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.209463    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-tls-certs\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.209665 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.209507    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-model-cache\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.310852 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.310820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-tmp-dir\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.310852 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.310855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-dshm\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.311106 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.310986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-home\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.311106 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.311029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-tls-certs\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.311106 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.311054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-model-cache\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.311106 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.311084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k8b8\" (UniqueName: \"kubernetes.io/projected/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-kube-api-access-8k8b8\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.311318 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.311261    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-tmp-dir\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.311374 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.311353    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-home\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.311415 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.311384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-model-cache\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.313352 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.313334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-dshm\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.313642 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.313620    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-tls-certs\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.319690 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.319669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k8b8\" (UniqueName: \"kubernetes.io/projected/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-kube-api-access-8k8b8\") pod \"auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") " pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.464218 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.464130    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:37:59.663432 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.663402    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:37:59.738302 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:37:59.738280    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"]
Sep 11 18:37:59.739837 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:37:59.739814    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa5bf91d_ab50_4628_98ad_0974d5d92b5a.slice/crio-7cb7d898f8483b9a031313d03baaaf7b65ebe02c8614b7ce52273a6df55bb080 WatchSource:0}: Error finding container 7cb7d898f8483b9a031313d03baaaf7b65ebe02c8614b7ce52273a6df55bb080: Status 404 returned error can't find the container with id 7cb7d898f8483b9a031313d03baaaf7b65ebe02c8614b7ce52273a6df55bb080
Sep 11 18:38:00.611574 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:38:00.611544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr" event={"ID":"aa5bf91d-ab50-4628-98ad-0974d5d92b5a","Type":"ContainerStarted","Data":"cec6ffd8f036a750c3380b791b9a0dabffde761b9484467ef716520bcdfb2da7"}
Sep 11 18:38:00.611936 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:38:00.611582    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr" event={"ID":"aa5bf91d-ab50-4628-98ad-0974d5d92b5a","Type":"ContainerStarted","Data":"7cb7d898f8483b9a031313d03baaaf7b65ebe02c8614b7ce52273a6df55bb080"}
Sep 11 18:38:00.637644 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:38:00.637589    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr" podStartSLOduration=1.637574888 podStartE2EDuration="1.637574888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:00.635365325 +0000 UTC m=+1357.376492169" watchObservedRunningTime="2026-09-11 18:38:00.637574888 +0000 UTC m=+1357.378701732"
Sep 11 18:38:09.465240 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:38:09.465204    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:38:09.465650 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:38:09.465262    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:38:09.478090 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:38:09.478056    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:38:09.671344 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:38:09.671320    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:38:32.143355 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:38:32.143316    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:39:10.252547 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:10.252504    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:39:42.424030 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.423998    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"]
Sep 11 18:39:42.427889 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.427869    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"
Sep 11 18:39:42.430086 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.430061    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-c1409b-dockercfg-m5lp7\""
Sep 11 18:39:42.430086 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.430075    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-c1409b\""
Sep 11 18:39:42.441352 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.441326    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"]
Sep 11 18:39:42.461622 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.461602    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"]
Sep 11 18:39:42.465173 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.465158    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"
Sep 11 18:39:42.474071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.474051    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"]
Sep 11 18:39:42.522294 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.522265    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19844962-8aff-4ceb-8df7-599523a170aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz\" (UID: \"19844962-8aff-4ceb-8df7-599523a170aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"
Sep 11 18:39:42.522422 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.522322    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7d4v\" (UniqueName: \"kubernetes.io/projected/19844962-8aff-4ceb-8df7-599523a170aa-kube-api-access-f7d4v\") pod \"payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz\" (UID: \"19844962-8aff-4ceb-8df7-599523a170aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"
Sep 11 18:39:42.623520 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.623464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a63389-fe8e-4888-9342-df7c93310e52-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg\" (UID: \"67a63389-fe8e-4888-9342-df7c93310e52\") " pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"
Sep 11 18:39:42.623637 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.623522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9j94\" (UniqueName: \"kubernetes.io/projected/67a63389-fe8e-4888-9342-df7c93310e52-kube-api-access-f9j94\") pod \"payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg\" (UID: \"67a63389-fe8e-4888-9342-df7c93310e52\") " pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"
Sep 11 18:39:42.623679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.623629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19844962-8aff-4ceb-8df7-599523a170aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz\" (UID: \"19844962-8aff-4ceb-8df7-599523a170aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"
Sep 11 18:39:42.623679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.623669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7d4v\" (UniqueName: \"kubernetes.io/projected/19844962-8aff-4ceb-8df7-599523a170aa-kube-api-access-f7d4v\") pod \"payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz\" (UID: \"19844962-8aff-4ceb-8df7-599523a170aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"
Sep 11 18:39:42.624244 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.624226    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19844962-8aff-4ceb-8df7-599523a170aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz\" (UID: \"19844962-8aff-4ceb-8df7-599523a170aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"
Sep 11 18:39:42.632779 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.632753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7d4v\" (UniqueName: \"kubernetes.io/projected/19844962-8aff-4ceb-8df7-599523a170aa-kube-api-access-f7d4v\") pod \"payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz\" (UID: \"19844962-8aff-4ceb-8df7-599523a170aa\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"
Sep 11 18:39:42.724506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.724435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a63389-fe8e-4888-9342-df7c93310e52-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg\" (UID: \"67a63389-fe8e-4888-9342-df7c93310e52\") " pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"
Sep 11 18:39:42.724506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.724463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9j94\" (UniqueName: \"kubernetes.io/projected/67a63389-fe8e-4888-9342-df7c93310e52-kube-api-access-f9j94\") pod \"payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg\" (UID: \"67a63389-fe8e-4888-9342-df7c93310e52\") " pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"
Sep 11 18:39:42.725043 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.725026    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a63389-fe8e-4888-9342-df7c93310e52-plugins-config-volume\") pod \"payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg\" (UID: \"67a63389-fe8e-4888-9342-df7c93310e52\") " pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"
Sep 11 18:39:42.732525 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.732493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9j94\" (UniqueName: \"kubernetes.io/projected/67a63389-fe8e-4888-9342-df7c93310e52-kube-api-access-f9j94\") pod \"payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg\" (UID: \"67a63389-fe8e-4888-9342-df7c93310e52\") " pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"
Sep 11 18:39:42.738281 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.738267    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"
Sep 11 18:39:42.776533 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:42.776258    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"
Sep 11 18:39:43.011983 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:43.011957    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"]
Sep 11 18:39:43.013215 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:39:43.013193    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19844962_8aff_4ceb_8df7_599523a170aa.slice/crio-b05dbc300d61b31e3b369297a7e97ec776b83db07d0ff3c49bf71599de70085a WatchSource:0}: Error finding container b05dbc300d61b31e3b369297a7e97ec776b83db07d0ff3c49bf71599de70085a: Status 404 returned error can't find the container with id b05dbc300d61b31e3b369297a7e97ec776b83db07d0ff3c49bf71599de70085a
Sep 11 18:39:43.067802 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:43.067779    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"]
Sep 11 18:39:43.069399 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:39:43.069375    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a63389_fe8e_4888_9342_df7c93310e52.slice/crio-b66c5dc6e37b136de4a4a2628bef77785750f454f0159d940175a7fcc1af47d8 WatchSource:0}: Error finding container b66c5dc6e37b136de4a4a2628bef77785750f454f0159d940175a7fcc1af47d8: Status 404 returned error can't find the container with id b66c5dc6e37b136de4a4a2628bef77785750f454f0159d940175a7fcc1af47d8
Sep 11 18:39:43.169967 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:43.169929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg" event={"ID":"67a63389-fe8e-4888-9342-df7c93310e52","Type":"ContainerStarted","Data":"af3736f5a100ace65e9f5ee6f75c3bc89b818bf13121ad030c37be4723d348a1"}
Sep 11 18:39:43.170157 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:43.169993    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg" event={"ID":"67a63389-fe8e-4888-9342-df7c93310e52","Type":"ContainerStarted","Data":"b66c5dc6e37b136de4a4a2628bef77785750f454f0159d940175a7fcc1af47d8"}
Sep 11 18:39:43.170157 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:43.170084    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"
Sep 11 18:39:43.171677 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:43.171649    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz" event={"ID":"19844962-8aff-4ceb-8df7-599523a170aa","Type":"ContainerStarted","Data":"e6efb4123e0952952353a00133280a2945204543f09b601fe37e75655ebc56e0"}
Sep 11 18:39:43.171793 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:43.171689    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz" event={"ID":"19844962-8aff-4ceb-8df7-599523a170aa","Type":"ContainerStarted","Data":"b05dbc300d61b31e3b369297a7e97ec776b83db07d0ff3c49bf71599de70085a"}
Sep 11 18:39:43.171793 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:43.171719    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"
Sep 11 18:39:43.187046 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:43.186997    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg" podStartSLOduration=1.186983776 podStartE2EDuration="1.186983776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:39:43.18489419 +0000 UTC m=+1459.926021033" watchObservedRunningTime="2026-09-11 18:39:43.186983776 +0000 UTC m=+1459.928110666"
Sep 11 18:39:43.201223 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:43.201178    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz" podStartSLOduration=1.201167143 podStartE2EDuration="1.201167143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:39:43.200559754 +0000 UTC m=+1459.941686599" watchObservedRunningTime="2026-09-11 18:39:43.201167143 +0000 UTC m=+1459.942293986"
Sep 11 18:39:44.948305 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:44.948272    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:39:54.179657 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:54.179630    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"
Sep 11 18:39:54.180066 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:39:54.179860    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"
Sep 11 18:40:13.018382 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:13.018349    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"]
Sep 11 18:40:13.022177 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:13.022157    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"
Sep 11 18:40:13.024552 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:13.024532    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ea789c-dockercfg-cwg5g\""
Sep 11 18:40:13.024798 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:13.024783    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ea789c\""
Sep 11 18:40:13.032234 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:13.032210    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"]
Sep 11 18:40:13.115258 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:13.115228    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56547d6a-6948-4873-81e2-c7df20fd6b0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq\" (UID: \"56547d6a-6948-4873-81e2-c7df20fd6b0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"
Sep 11 18:40:13.115431 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:13.115291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ddhn\" (UniqueName: \"kubernetes.io/projected/56547d6a-6948-4873-81e2-c7df20fd6b0e-kube-api-access-5ddhn\") pod \"payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq\" (UID: \"56547d6a-6948-4873-81e2-c7df20fd6b0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"
Sep 11 18:40:13.216558 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:13.216522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56547d6a-6948-4873-81e2-c7df20fd6b0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq\" (UID: \"56547d6a-6948-4873-81e2-c7df20fd6b0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"
Sep 11 18:40:13.216769 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:13.216582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ddhn\" (UniqueName: \"kubernetes.io/projected/56547d6a-6948-4873-81e2-c7df20fd6b0e-kube-api-access-5ddhn\") pod \"payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq\" (UID: \"56547d6a-6948-4873-81e2-c7df20fd6b0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"
Sep 11 18:40:13.217176 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:13.217156    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56547d6a-6948-4873-81e2-c7df20fd6b0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq\" (UID: \"56547d6a-6948-4873-81e2-c7df20fd6b0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"
Sep 11 18:40:13.228824 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:13.228801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ddhn\" (UniqueName: \"kubernetes.io/projected/56547d6a-6948-4873-81e2-c7df20fd6b0e-kube-api-access-5ddhn\") pod \"payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq\" (UID: \"56547d6a-6948-4873-81e2-c7df20fd6b0e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"
Sep 11 18:40:13.334518 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:13.334420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"
Sep 11 18:40:13.605895 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:13.605872    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"]
Sep 11 18:40:13.607507 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:40:13.607461    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56547d6a_6948_4873_81e2_c7df20fd6b0e.slice/crio-775928c26f8d2b58161a1750c755c9f581ac382808e1964a2611fa00d8f239f4 WatchSource:0}: Error finding container 775928c26f8d2b58161a1750c755c9f581ac382808e1964a2611fa00d8f239f4: Status 404 returned error can't find the container with id 775928c26f8d2b58161a1750c755c9f581ac382808e1964a2611fa00d8f239f4
Sep 11 18:40:14.352579 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:14.352530    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq" event={"ID":"56547d6a-6948-4873-81e2-c7df20fd6b0e","Type":"ContainerStarted","Data":"1be56805ed12c9932135369589e2c9daf52bc8f83cc9ecc616679c15a7c2c38f"}
Sep 11 18:40:14.353032 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:14.352609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq" event={"ID":"56547d6a-6948-4873-81e2-c7df20fd6b0e","Type":"ContainerStarted","Data":"775928c26f8d2b58161a1750c755c9f581ac382808e1964a2611fa00d8f239f4"}
Sep 11 18:40:14.353032 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:14.352646    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"
Sep 11 18:40:14.370981 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:14.370937    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq" podStartSLOduration=2.370923137 podStartE2EDuration="2.370923137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:40:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:40:14.368126656 +0000 UTC m=+1491.109253523" watchObservedRunningTime="2026-09-11 18:40:14.370923137 +0000 UTC m=+1491.112049980"
Sep 11 18:40:25.360336 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:25.360301    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"
Sep 11 18:40:31.550923 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:31.550891    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:40:37.992895 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:37.992859    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt"]
Sep 11 18:40:37.996654 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:37.996636    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt"
Sep 11 18:40:37.998723 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:37.998699    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:40:37.998818 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:37.998733    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\""
Sep 11 18:40:38.005124 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:38.005095    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt"]
Sep 11 18:40:38.050455 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:38.050406    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1477908f-c86f-41fd-b7f0-4d6b4cb32b60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt\" (UID: \"1477908f-c86f-41fd-b7f0-4d6b4cb32b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt"
Sep 11 18:40:38.151693 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:38.151661    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1477908f-c86f-41fd-b7f0-4d6b4cb32b60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt\" (UID: \"1477908f-c86f-41fd-b7f0-4d6b4cb32b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt"
Sep 11 18:40:38.152283 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:38.152259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1477908f-c86f-41fd-b7f0-4d6b4cb32b60-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt\" (UID: \"1477908f-c86f-41fd-b7f0-4d6b4cb32b60\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt"
Sep 11 18:40:38.308870 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:38.308794    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt"
Sep 11 18:40:38.581051 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:38.581024    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt"]
Sep 11 18:40:38.582737 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:40:38.582709    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1477908f_c86f_41fd_b7f0_4d6b4cb32b60.slice/crio-19cb88773440cf2e40535c9b5706a6b245c55ad2212aedcd0472f64cb9c7d281 WatchSource:0}: Error finding container 19cb88773440cf2e40535c9b5706a6b245c55ad2212aedcd0472f64cb9c7d281: Status 404 returned error can't find the container with id 19cb88773440cf2e40535c9b5706a6b245c55ad2212aedcd0472f64cb9c7d281
Sep 11 18:40:39.494151 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:39.494058    2581 generic.go:356] "Generic (PLEG): container finished" podID="1477908f-c86f-41fd-b7f0-4d6b4cb32b60" containerID="9b95e3a367e6bd1a7cdd6092bca73a2a4efc46d8879f1af924948d645f67ad8c" exitCode=0
Sep 11 18:40:39.494151 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:39.494108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt" event={"ID":"1477908f-c86f-41fd-b7f0-4d6b4cb32b60","Type":"ContainerDied","Data":"9b95e3a367e6bd1a7cdd6092bca73a2a4efc46d8879f1af924948d645f67ad8c"}
Sep 11 18:40:39.494552 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:39.494159    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt" event={"ID":"1477908f-c86f-41fd-b7f0-4d6b4cb32b60","Type":"ContainerStarted","Data":"19cb88773440cf2e40535c9b5706a6b245c55ad2212aedcd0472f64cb9c7d281"}
Sep 11 18:40:40.641199 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:40.641175    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt"
Sep 11 18:40:40.778847 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:40.778766    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1477908f-c86f-41fd-b7f0-4d6b4cb32b60-maas-api-service-ca\") pod \"1477908f-c86f-41fd-b7f0-4d6b4cb32b60\" (UID: \"1477908f-c86f-41fd-b7f0-4d6b4cb32b60\") "
Sep 11 18:40:40.779104 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:40.779081    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1477908f-c86f-41fd-b7f0-4d6b4cb32b60-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1477908f-c86f-41fd-b7f0-4d6b4cb32b60" (UID: "1477908f-c86f-41fd-b7f0-4d6b4cb32b60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:40:40.880276 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:40.880237    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1477908f-c86f-41fd-b7f0-4d6b4cb32b60-maas-api-service-ca\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:40:41.507262 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:41.507230    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt" event={"ID":"1477908f-c86f-41fd-b7f0-4d6b4cb32b60","Type":"ContainerDied","Data":"19cb88773440cf2e40535c9b5706a6b245c55ad2212aedcd0472f64cb9c7d281"}
Sep 11 18:40:41.507438 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:41.507278    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19cb88773440cf2e40535c9b5706a6b245c55ad2212aedcd0472f64cb9c7d281"
Sep 11 18:40:41.507438 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:41.507244    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt"
Sep 11 18:40:54.074086 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.074003    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"]
Sep 11 18:40:54.074640 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.074274    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg" podUID="67a63389-fe8e-4888-9342-df7c93310e52" containerName="payload-processing" containerID="cri-o://af3736f5a100ace65e9f5ee6f75c3bc89b818bf13121ad030c37be4723d348a1" gracePeriod=30
Sep 11 18:40:54.082667 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.082642    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"]
Sep 11 18:40:54.082907 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.082885    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz" podUID="19844962-8aff-4ceb-8df7-599523a170aa" containerName="payload-pre-processing" containerID="cri-o://e6efb4123e0952952353a00133280a2945204543f09b601fe37e75655ebc56e0" gracePeriod=30
Sep 11 18:40:54.177660 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.177603    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg" podUID="67a63389-fe8e-4888-9342-df7c93310e52" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.76:9004: connect: connection refused"
Sep 11 18:40:54.177803 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.177606    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz" podUID="19844962-8aff-4ceb-8df7-599523a170aa" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.75:9004: connect: connection refused"
Sep 11 18:40:54.270904 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.270875    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt"]
Sep 11 18:40:54.273979 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.273953    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c1409b-xkjrt"]
Sep 11 18:40:54.365666 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.365641    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"
Sep 11 18:40:54.368551 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.368503    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"
Sep 11 18:40:54.521692 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.521661    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a63389-fe8e-4888-9342-df7c93310e52-plugins-config-volume\") pod \"67a63389-fe8e-4888-9342-df7c93310e52\" (UID: \"67a63389-fe8e-4888-9342-df7c93310e52\") "
Sep 11 18:40:54.521886 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.521738    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7d4v\" (UniqueName: \"kubernetes.io/projected/19844962-8aff-4ceb-8df7-599523a170aa-kube-api-access-f7d4v\") pod \"19844962-8aff-4ceb-8df7-599523a170aa\" (UID: \"19844962-8aff-4ceb-8df7-599523a170aa\") "
Sep 11 18:40:54.521886 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.521808    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19844962-8aff-4ceb-8df7-599523a170aa-plugins-config-volume\") pod \"19844962-8aff-4ceb-8df7-599523a170aa\" (UID: \"19844962-8aff-4ceb-8df7-599523a170aa\") "
Sep 11 18:40:54.521886 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.521830    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9j94\" (UniqueName: \"kubernetes.io/projected/67a63389-fe8e-4888-9342-df7c93310e52-kube-api-access-f9j94\") pod \"67a63389-fe8e-4888-9342-df7c93310e52\" (UID: \"67a63389-fe8e-4888-9342-df7c93310e52\") "
Sep 11 18:40:54.522143 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.522118    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67a63389-fe8e-4888-9342-df7c93310e52-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67a63389-fe8e-4888-9342-df7c93310e52" (UID: "67a63389-fe8e-4888-9342-df7c93310e52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:40:54.522230 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.522203    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19844962-8aff-4ceb-8df7-599523a170aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19844962-8aff-4ceb-8df7-599523a170aa" (UID: "19844962-8aff-4ceb-8df7-599523a170aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:40:54.522280 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.522222    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67a63389-fe8e-4888-9342-df7c93310e52-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:40:54.524013 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.523989    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19844962-8aff-4ceb-8df7-599523a170aa-kube-api-access-f7d4v" (OuterVolumeSpecName: "kube-api-access-f7d4v") pod "19844962-8aff-4ceb-8df7-599523a170aa" (UID: "19844962-8aff-4ceb-8df7-599523a170aa"). InnerVolumeSpecName "kube-api-access-f7d4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:40:54.524093 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.524026    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a63389-fe8e-4888-9342-df7c93310e52-kube-api-access-f9j94" (OuterVolumeSpecName: "kube-api-access-f9j94") pod "67a63389-fe8e-4888-9342-df7c93310e52" (UID: "67a63389-fe8e-4888-9342-df7c93310e52"). InnerVolumeSpecName "kube-api-access-f9j94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:40:54.579548 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.579512    2581 generic.go:356] "Generic (PLEG): container finished" podID="67a63389-fe8e-4888-9342-df7c93310e52" containerID="af3736f5a100ace65e9f5ee6f75c3bc89b818bf13121ad030c37be4723d348a1" exitCode=0
Sep 11 18:40:54.579720 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.579550    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"
Sep 11 18:40:54.579720 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.579589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg" event={"ID":"67a63389-fe8e-4888-9342-df7c93310e52","Type":"ContainerDied","Data":"af3736f5a100ace65e9f5ee6f75c3bc89b818bf13121ad030c37be4723d348a1"}
Sep 11 18:40:54.579720 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.579686    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg" event={"ID":"67a63389-fe8e-4888-9342-df7c93310e52","Type":"ContainerDied","Data":"b66c5dc6e37b136de4a4a2628bef77785750f454f0159d940175a7fcc1af47d8"}
Sep 11 18:40:54.579720 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.579714    2581 scope.go:117] "RemoveContainer" containerID="af3736f5a100ace65e9f5ee6f75c3bc89b818bf13121ad030c37be4723d348a1"
Sep 11 18:40:54.581178 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.581154    2581 generic.go:356] "Generic (PLEG): container finished" podID="19844962-8aff-4ceb-8df7-599523a170aa" containerID="e6efb4123e0952952353a00133280a2945204543f09b601fe37e75655ebc56e0" exitCode=0
Sep 11 18:40:54.581316 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.581198    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"
Sep 11 18:40:54.581316 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.581236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz" event={"ID":"19844962-8aff-4ceb-8df7-599523a170aa","Type":"ContainerDied","Data":"e6efb4123e0952952353a00133280a2945204543f09b601fe37e75655ebc56e0"}
Sep 11 18:40:54.581316 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.581311    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz" event={"ID":"19844962-8aff-4ceb-8df7-599523a170aa","Type":"ContainerDied","Data":"b05dbc300d61b31e3b369297a7e97ec776b83db07d0ff3c49bf71599de70085a"}
Sep 11 18:40:54.590166 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.590021    2581 scope.go:117] "RemoveContainer" containerID="af3736f5a100ace65e9f5ee6f75c3bc89b818bf13121ad030c37be4723d348a1"
Sep 11 18:40:54.590586 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:40:54.590559    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af3736f5a100ace65e9f5ee6f75c3bc89b818bf13121ad030c37be4723d348a1\": container with ID starting with af3736f5a100ace65e9f5ee6f75c3bc89b818bf13121ad030c37be4723d348a1 not found: ID does not exist" containerID="af3736f5a100ace65e9f5ee6f75c3bc89b818bf13121ad030c37be4723d348a1"
Sep 11 18:40:54.590698 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.590587    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3736f5a100ace65e9f5ee6f75c3bc89b818bf13121ad030c37be4723d348a1"} err="failed to get container status \"af3736f5a100ace65e9f5ee6f75c3bc89b818bf13121ad030c37be4723d348a1\": rpc error: code = NotFound desc = could not find container \"af3736f5a100ace65e9f5ee6f75c3bc89b818bf13121ad030c37be4723d348a1\": container with ID starting with af3736f5a100ace65e9f5ee6f75c3bc89b818bf13121ad030c37be4723d348a1 not found: ID does not exist"
Sep 11 18:40:54.590698 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.590604    2581 scope.go:117] "RemoveContainer" containerID="e6efb4123e0952952353a00133280a2945204543f09b601fe37e75655ebc56e0"
Sep 11 18:40:54.599222 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.599206    2581 scope.go:117] "RemoveContainer" containerID="e6efb4123e0952952353a00133280a2945204543f09b601fe37e75655ebc56e0"
Sep 11 18:40:54.599501 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:40:54.599456    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6efb4123e0952952353a00133280a2945204543f09b601fe37e75655ebc56e0\": container with ID starting with e6efb4123e0952952353a00133280a2945204543f09b601fe37e75655ebc56e0 not found: ID does not exist" containerID="e6efb4123e0952952353a00133280a2945204543f09b601fe37e75655ebc56e0"
Sep 11 18:40:54.599598 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.599581    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6efb4123e0952952353a00133280a2945204543f09b601fe37e75655ebc56e0"} err="failed to get container status \"e6efb4123e0952952353a00133280a2945204543f09b601fe37e75655ebc56e0\": rpc error: code = NotFound desc = could not find container \"e6efb4123e0952952353a00133280a2945204543f09b601fe37e75655ebc56e0\": container with ID starting with e6efb4123e0952952353a00133280a2945204543f09b601fe37e75655ebc56e0 not found: ID does not exist"
Sep 11 18:40:54.608603 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.608582    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"]
Sep 11 18:40:54.613165 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.613145    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-c1409b-7f5b7d969-sf8pg"]
Sep 11 18:40:54.623145 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.623123    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7d4v\" (UniqueName: \"kubernetes.io/projected/19844962-8aff-4ceb-8df7-599523a170aa-kube-api-access-f7d4v\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:40:54.623145 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.623144    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19844962-8aff-4ceb-8df7-599523a170aa-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:40:54.623243 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.623153    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9j94\" (UniqueName: \"kubernetes.io/projected/67a63389-fe8e-4888-9342-df7c93310e52-kube-api-access-f9j94\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:40:54.628388 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.628367    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"]
Sep 11 18:40:54.630437 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:54.630413    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-c1409b-55bd6d49d-tswjz"]
Sep 11 18:40:55.764199 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:55.764167    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1477908f-c86f-41fd-b7f0-4d6b4cb32b60" path="/var/lib/kubelet/pods/1477908f-c86f-41fd-b7f0-4d6b4cb32b60/volumes"
Sep 11 18:40:55.764562 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:55.764552    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19844962-8aff-4ceb-8df7-599523a170aa" path="/var/lib/kubelet/pods/19844962-8aff-4ceb-8df7-599523a170aa/volumes"
Sep 11 18:40:55.764892 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:40:55.764878    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67a63389-fe8e-4888-9342-df7c93310e52" path="/var/lib/kubelet/pods/67a63389-fe8e-4888-9342-df7c93310e52/volumes"
Sep 11 18:41:01.556166 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.556129    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv"]
Sep 11 18:41:01.556624 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.556610    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a63389-fe8e-4888-9342-df7c93310e52" containerName="payload-processing"
Sep 11 18:41:01.556687 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.556626    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a63389-fe8e-4888-9342-df7c93310e52" containerName="payload-processing"
Sep 11 18:41:01.556687 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.556638    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19844962-8aff-4ceb-8df7-599523a170aa" containerName="payload-pre-processing"
Sep 11 18:41:01.556687 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.556644    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="19844962-8aff-4ceb-8df7-599523a170aa" containerName="payload-pre-processing"
Sep 11 18:41:01.556687 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.556657    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1477908f-c86f-41fd-b7f0-4d6b4cb32b60" containerName="revoke-keys"
Sep 11 18:41:01.556687 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.556663    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1477908f-c86f-41fd-b7f0-4d6b4cb32b60" containerName="revoke-keys"
Sep 11 18:41:01.556867 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.556734    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a63389-fe8e-4888-9342-df7c93310e52" containerName="payload-processing"
Sep 11 18:41:01.556867 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.556742    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="19844962-8aff-4ceb-8df7-599523a170aa" containerName="payload-pre-processing"
Sep 11 18:41:01.556867 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.556750    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1477908f-c86f-41fd-b7f0-4d6b4cb32b60" containerName="revoke-keys"
Sep 11 18:41:01.561453 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.561436    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv"
Sep 11 18:41:01.563550 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.563531    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:41:01.563674 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.563531    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\""
Sep 11 18:41:01.568809 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.568781    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv"]
Sep 11 18:41:01.587145 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.587120    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b08591-6c44-47e2-8f60-9ee8b4c9edd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv\" (UID: \"f7b08591-6c44-47e2-8f60-9ee8b4c9edd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv"
Sep 11 18:41:01.687872 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.687844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b08591-6c44-47e2-8f60-9ee8b4c9edd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv\" (UID: \"f7b08591-6c44-47e2-8f60-9ee8b4c9edd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv"
Sep 11 18:41:01.688406 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.688387    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b08591-6c44-47e2-8f60-9ee8b4c9edd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv\" (UID: \"f7b08591-6c44-47e2-8f60-9ee8b4c9edd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv"
Sep 11 18:41:01.874770 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:01.874689    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv"
Sep 11 18:41:02.151995 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:02.151970    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv"]
Sep 11 18:41:02.153550 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:41:02.153523    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b08591_6c44_47e2_8f60_9ee8b4c9edd5.slice/crio-b9dfcee176888eaafc18cb33eab548d4edccab905f62c7bc82e64b6a811856ce WatchSource:0}: Error finding container b9dfcee176888eaafc18cb33eab548d4edccab905f62c7bc82e64b6a811856ce: Status 404 returned error can't find the container with id b9dfcee176888eaafc18cb33eab548d4edccab905f62c7bc82e64b6a811856ce
Sep 11 18:41:02.630249 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:02.630218    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv" event={"ID":"f7b08591-6c44-47e2-8f60-9ee8b4c9edd5","Type":"ContainerStarted","Data":"25a1a0d932955197ef54a91f41ce8c6729a5cd207d374525912092733fed5b37"}
Sep 11 18:41:02.630620 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:02.630268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv" event={"ID":"f7b08591-6c44-47e2-8f60-9ee8b4c9edd5","Type":"ContainerStarted","Data":"b9dfcee176888eaafc18cb33eab548d4edccab905f62c7bc82e64b6a811856ce"}
Sep 11 18:41:02.646097 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:02.646049    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv" podStartSLOduration=1.646034593 podStartE2EDuration="1.646034593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:41:02.643852001 +0000 UTC m=+1539.384978845" watchObservedRunningTime="2026-09-11 18:41:02.646034593 +0000 UTC m=+1539.387161603"
Sep 11 18:41:03.637573 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:03.637534    2581 generic.go:356] "Generic (PLEG): container finished" podID="f7b08591-6c44-47e2-8f60-9ee8b4c9edd5" containerID="25a1a0d932955197ef54a91f41ce8c6729a5cd207d374525912092733fed5b37" exitCode=0
Sep 11 18:41:03.637968 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:03.637629    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv" event={"ID":"f7b08591-6c44-47e2-8f60-9ee8b4c9edd5","Type":"ContainerDied","Data":"25a1a0d932955197ef54a91f41ce8c6729a5cd207d374525912092733fed5b37"}
Sep 11 18:41:04.781650 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:04.781626    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv"
Sep 11 18:41:04.820603 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:04.820571    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b08591-6c44-47e2-8f60-9ee8b4c9edd5-maas-api-service-ca\") pod \"f7b08591-6c44-47e2-8f60-9ee8b4c9edd5\" (UID: \"f7b08591-6c44-47e2-8f60-9ee8b4c9edd5\") "
Sep 11 18:41:04.820897 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:04.820873    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b08591-6c44-47e2-8f60-9ee8b4c9edd5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7b08591-6c44-47e2-8f60-9ee8b4c9edd5" (UID: "f7b08591-6c44-47e2-8f60-9ee8b4c9edd5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:41:04.821025 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:04.821008    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b08591-6c44-47e2-8f60-9ee8b4c9edd5-maas-api-service-ca\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:05.650073 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:05.650042    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv"
Sep 11 18:41:05.650073 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:05.650051    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv" event={"ID":"f7b08591-6c44-47e2-8f60-9ee8b4c9edd5","Type":"ContainerDied","Data":"b9dfcee176888eaafc18cb33eab548d4edccab905f62c7bc82e64b6a811856ce"}
Sep 11 18:41:05.650281 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:05.650099    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9dfcee176888eaafc18cb33eab548d4edccab905f62c7bc82e64b6a811856ce"
Sep 11 18:41:17.396269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.396236    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"]
Sep 11 18:41:17.396759 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.396514    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq" podUID="56547d6a-6948-4873-81e2-c7df20fd6b0e" containerName="payload-pre-processing" containerID="cri-o://1be56805ed12c9932135369589e2c9daf52bc8f83cc9ecc616679c15a7c2c38f" gracePeriod=30
Sep 11 18:41:17.592463 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.592428    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv"]
Sep 11 18:41:17.595631 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.595603    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea789c-sw2jv"]
Sep 11 18:41:17.668521 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.668469    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"
Sep 11 18:41:17.721683 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.721651    2581 generic.go:356] "Generic (PLEG): container finished" podID="56547d6a-6948-4873-81e2-c7df20fd6b0e" containerID="1be56805ed12c9932135369589e2c9daf52bc8f83cc9ecc616679c15a7c2c38f" exitCode=0
Sep 11 18:41:17.721830 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.721694    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"
Sep 11 18:41:17.721830 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.721728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq" event={"ID":"56547d6a-6948-4873-81e2-c7df20fd6b0e","Type":"ContainerDied","Data":"1be56805ed12c9932135369589e2c9daf52bc8f83cc9ecc616679c15a7c2c38f"}
Sep 11 18:41:17.721830 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.721779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq" event={"ID":"56547d6a-6948-4873-81e2-c7df20fd6b0e","Type":"ContainerDied","Data":"775928c26f8d2b58161a1750c755c9f581ac382808e1964a2611fa00d8f239f4"}
Sep 11 18:41:17.721830 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.721802    2581 scope.go:117] "RemoveContainer" containerID="1be56805ed12c9932135369589e2c9daf52bc8f83cc9ecc616679c15a7c2c38f"
Sep 11 18:41:17.730907 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.730889    2581 scope.go:117] "RemoveContainer" containerID="1be56805ed12c9932135369589e2c9daf52bc8f83cc9ecc616679c15a7c2c38f"
Sep 11 18:41:17.731171 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:41:17.731152    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be56805ed12c9932135369589e2c9daf52bc8f83cc9ecc616679c15a7c2c38f\": container with ID starting with 1be56805ed12c9932135369589e2c9daf52bc8f83cc9ecc616679c15a7c2c38f not found: ID does not exist" containerID="1be56805ed12c9932135369589e2c9daf52bc8f83cc9ecc616679c15a7c2c38f"
Sep 11 18:41:17.731218 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.731178    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be56805ed12c9932135369589e2c9daf52bc8f83cc9ecc616679c15a7c2c38f"} err="failed to get container status \"1be56805ed12c9932135369589e2c9daf52bc8f83cc9ecc616679c15a7c2c38f\": rpc error: code = NotFound desc = could not find container \"1be56805ed12c9932135369589e2c9daf52bc8f83cc9ecc616679c15a7c2c38f\": container with ID starting with 1be56805ed12c9932135369589e2c9daf52bc8f83cc9ecc616679c15a7c2c38f not found: ID does not exist"
Sep 11 18:41:17.755284 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.755249    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ddhn\" (UniqueName: \"kubernetes.io/projected/56547d6a-6948-4873-81e2-c7df20fd6b0e-kube-api-access-5ddhn\") pod \"56547d6a-6948-4873-81e2-c7df20fd6b0e\" (UID: \"56547d6a-6948-4873-81e2-c7df20fd6b0e\") "
Sep 11 18:41:17.755409 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.755307    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56547d6a-6948-4873-81e2-c7df20fd6b0e-plugins-config-volume\") pod \"56547d6a-6948-4873-81e2-c7df20fd6b0e\" (UID: \"56547d6a-6948-4873-81e2-c7df20fd6b0e\") "
Sep 11 18:41:17.755770 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.755736    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56547d6a-6948-4873-81e2-c7df20fd6b0e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56547d6a-6948-4873-81e2-c7df20fd6b0e" (UID: "56547d6a-6948-4873-81e2-c7df20fd6b0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:41:17.757331 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.757300    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56547d6a-6948-4873-81e2-c7df20fd6b0e-kube-api-access-5ddhn" (OuterVolumeSpecName: "kube-api-access-5ddhn") pod "56547d6a-6948-4873-81e2-c7df20fd6b0e" (UID: "56547d6a-6948-4873-81e2-c7df20fd6b0e"). InnerVolumeSpecName "kube-api-access-5ddhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:17.765679 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.765654    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7b08591-6c44-47e2-8f60-9ee8b4c9edd5" path="/var/lib/kubelet/pods/f7b08591-6c44-47e2-8f60-9ee8b4c9edd5/volumes"
Sep 11 18:41:17.856996 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.856956    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ddhn\" (UniqueName: \"kubernetes.io/projected/56547d6a-6948-4873-81e2-c7df20fd6b0e-kube-api-access-5ddhn\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:17.856996 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:17.856994    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56547d6a-6948-4873-81e2-c7df20fd6b0e-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:18.038589 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:18.038506    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"]
Sep 11 18:41:18.043001 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:18.042977    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ea789c-67f66c95f4-k5vnq"]
Sep 11 18:41:19.765005 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:19.764969    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56547d6a-6948-4873-81e2-c7df20fd6b0e" path="/var/lib/kubelet/pods/56547d6a-6948-4873-81e2-c7df20fd6b0e/volumes"
Sep 11 18:41:24.251390 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:24.251302    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:41:24.713709 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:24.713677    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps"]
Sep 11 18:41:24.714114 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:24.714103    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56547d6a-6948-4873-81e2-c7df20fd6b0e" containerName="payload-pre-processing"
Sep 11 18:41:24.714160 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:24.714117    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="56547d6a-6948-4873-81e2-c7df20fd6b0e" containerName="payload-pre-processing"
Sep 11 18:41:24.714160 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:24.714143    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7b08591-6c44-47e2-8f60-9ee8b4c9edd5" containerName="revoke-keys"
Sep 11 18:41:24.714160 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:24.714149    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7b08591-6c44-47e2-8f60-9ee8b4c9edd5" containerName="revoke-keys"
Sep 11 18:41:24.714261 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:24.714211    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="56547d6a-6948-4873-81e2-c7df20fd6b0e" containerName="payload-pre-processing"
Sep 11 18:41:24.714261 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:24.714221    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7b08591-6c44-47e2-8f60-9ee8b4c9edd5" containerName="revoke-keys"
Sep 11 18:41:24.718675 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:24.718659    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps"
Sep 11 18:41:24.720803 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:24.720771    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\""
Sep 11 18:41:24.720933 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:24.720857    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:41:24.725253 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:24.725231    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps"]
Sep 11 18:41:24.823222 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:24.823186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d3ff081-928d-40a8-b3aa-56160cc780ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps\" (UID: \"7d3ff081-928d-40a8-b3aa-56160cc780ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps"
Sep 11 18:41:24.924012 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:24.923978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d3ff081-928d-40a8-b3aa-56160cc780ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps\" (UID: \"7d3ff081-928d-40a8-b3aa-56160cc780ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps"
Sep 11 18:41:24.924706 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:24.924687    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d3ff081-928d-40a8-b3aa-56160cc780ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps\" (UID: \"7d3ff081-928d-40a8-b3aa-56160cc780ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps"
Sep 11 18:41:25.030344 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:25.030278    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps"
Sep 11 18:41:25.302462 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:25.302435    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps"]
Sep 11 18:41:25.304077 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:41:25.304050    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d3ff081_928d_40a8_b3aa_56160cc780ee.slice/crio-a02837ecf9a82907a548ff5c7767f74079a62fd128d54851220c10fc8faee81e WatchSource:0}: Error finding container a02837ecf9a82907a548ff5c7767f74079a62fd128d54851220c10fc8faee81e: Status 404 returned error can't find the container with id a02837ecf9a82907a548ff5c7767f74079a62fd128d54851220c10fc8faee81e
Sep 11 18:41:25.768728 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:25.768700    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps" event={"ID":"7d3ff081-928d-40a8-b3aa-56160cc780ee","Type":"ContainerStarted","Data":"d4ce4ced49a87baee373cba9ae6bc34b372bf16124fbcfc6d1fb6a428423c7d0"}
Sep 11 18:41:25.768882 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:25.768746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps" event={"ID":"7d3ff081-928d-40a8-b3aa-56160cc780ee","Type":"ContainerStarted","Data":"a02837ecf9a82907a548ff5c7767f74079a62fd128d54851220c10fc8faee81e"}
Sep 11 18:41:25.784447 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:25.784399    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps" podStartSLOduration=1.784384526 podStartE2EDuration="1.784384526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:41:25.783617371 +0000 UTC m=+1562.524744227" watchObservedRunningTime="2026-09-11 18:41:25.784384526 +0000 UTC m=+1562.525511417"
Sep 11 18:41:26.775470 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:26.775431    2581 generic.go:356] "Generic (PLEG): container finished" podID="7d3ff081-928d-40a8-b3aa-56160cc780ee" containerID="d4ce4ced49a87baee373cba9ae6bc34b372bf16124fbcfc6d1fb6a428423c7d0" exitCode=0
Sep 11 18:41:26.775905 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:26.775499    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps" event={"ID":"7d3ff081-928d-40a8-b3aa-56160cc780ee","Type":"ContainerDied","Data":"d4ce4ced49a87baee373cba9ae6bc34b372bf16124fbcfc6d1fb6a428423c7d0"}
Sep 11 18:41:27.922689 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:27.922666    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps"
Sep 11 18:41:28.055343 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:28.055264    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d3ff081-928d-40a8-b3aa-56160cc780ee-maas-api-service-ca\") pod \"7d3ff081-928d-40a8-b3aa-56160cc780ee\" (UID: \"7d3ff081-928d-40a8-b3aa-56160cc780ee\") "
Sep 11 18:41:28.055627 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:28.055606    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d3ff081-928d-40a8-b3aa-56160cc780ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7d3ff081-928d-40a8-b3aa-56160cc780ee" (UID: "7d3ff081-928d-40a8-b3aa-56160cc780ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:41:28.156579 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:28.156537    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d3ff081-928d-40a8-b3aa-56160cc780ee-maas-api-service-ca\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:28.787013 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:28.786984    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps"
Sep 11 18:41:28.787180 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:28.787019    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps" event={"ID":"7d3ff081-928d-40a8-b3aa-56160cc780ee","Type":"ContainerDied","Data":"a02837ecf9a82907a548ff5c7767f74079a62fd128d54851220c10fc8faee81e"}
Sep 11 18:41:28.787180 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:28.787059    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a02837ecf9a82907a548ff5c7767f74079a62fd128d54851220c10fc8faee81e"
Sep 11 18:41:31.878108 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:31.878072    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk"]
Sep 11 18:41:31.878542 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:31.878527    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d3ff081-928d-40a8-b3aa-56160cc780ee" containerName="revoke-keys"
Sep 11 18:41:31.878594 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:31.878545    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d3ff081-928d-40a8-b3aa-56160cc780ee" containerName="revoke-keys"
Sep 11 18:41:31.878646 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:31.878637    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d3ff081-928d-40a8-b3aa-56160cc780ee" containerName="revoke-keys"
Sep 11 18:41:31.881720 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:31.881703    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk"
Sep 11 18:41:31.883666 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:31.883646    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\""
Sep 11 18:41:31.883765 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:31.883690    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:41:31.888214 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:31.888193    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk"]
Sep 11 18:41:31.989906 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:31.989875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/393a4c1e-cca3-4ff4-a31a-6d09d44dbb01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk\" (UID: \"393a4c1e-cca3-4ff4-a31a-6d09d44dbb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk"
Sep 11 18:41:32.090822 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:32.090787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/393a4c1e-cca3-4ff4-a31a-6d09d44dbb01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk\" (UID: \"393a4c1e-cca3-4ff4-a31a-6d09d44dbb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk"
Sep 11 18:41:32.091333 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:32.091315    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/393a4c1e-cca3-4ff4-a31a-6d09d44dbb01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk\" (UID: \"393a4c1e-cca3-4ff4-a31a-6d09d44dbb01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk"
Sep 11 18:41:32.193240 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:32.193178    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk"
Sep 11 18:41:32.466800 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:32.466776    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk"]
Sep 11 18:41:32.469721 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:41:32.469687    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393a4c1e_cca3_4ff4_a31a_6d09d44dbb01.slice/crio-c82b815419d1e36fbd961ec8faa7c56c6885c74bfce1da8aa0e5de54c4203a17 WatchSource:0}: Error finding container c82b815419d1e36fbd961ec8faa7c56c6885c74bfce1da8aa0e5de54c4203a17: Status 404 returned error can't find the container with id c82b815419d1e36fbd961ec8faa7c56c6885c74bfce1da8aa0e5de54c4203a17
Sep 11 18:41:32.810382 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:32.810350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk" event={"ID":"393a4c1e-cca3-4ff4-a31a-6d09d44dbb01","Type":"ContainerStarted","Data":"0e92a56342b6e2968e9e92a2e48dbf8b9dc6821f89359c8663d07e5af3a02721"}
Sep 11 18:41:32.810534 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:32.810395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk" event={"ID":"393a4c1e-cca3-4ff4-a31a-6d09d44dbb01","Type":"ContainerStarted","Data":"c82b815419d1e36fbd961ec8faa7c56c6885c74bfce1da8aa0e5de54c4203a17"}
Sep 11 18:41:32.827296 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:32.827256    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk" podStartSLOduration=1.827244249 podStartE2EDuration="1.827244249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:41:32.82638116 +0000 UTC m=+1569.567508003" watchObservedRunningTime="2026-09-11 18:41:32.827244249 +0000 UTC m=+1569.568371088"
Sep 11 18:41:33.816825 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:33.816792    2581 generic.go:356] "Generic (PLEG): container finished" podID="393a4c1e-cca3-4ff4-a31a-6d09d44dbb01" containerID="0e92a56342b6e2968e9e92a2e48dbf8b9dc6821f89359c8663d07e5af3a02721" exitCode=0
Sep 11 18:41:33.817202 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:33.816882    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk" event={"ID":"393a4c1e-cca3-4ff4-a31a-6d09d44dbb01","Type":"ContainerDied","Data":"0e92a56342b6e2968e9e92a2e48dbf8b9dc6821f89359c8663d07e5af3a02721"}
Sep 11 18:41:34.961513 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:34.961472    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk"
Sep 11 18:41:35.122625 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.122539    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/393a4c1e-cca3-4ff4-a31a-6d09d44dbb01-maas-api-service-ca\") pod \"393a4c1e-cca3-4ff4-a31a-6d09d44dbb01\" (UID: \"393a4c1e-cca3-4ff4-a31a-6d09d44dbb01\") "
Sep 11 18:41:35.122945 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.122921    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393a4c1e-cca3-4ff4-a31a-6d09d44dbb01-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "393a4c1e-cca3-4ff4-a31a-6d09d44dbb01" (UID: "393a4c1e-cca3-4ff4-a31a-6d09d44dbb01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:41:35.223683 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.223647    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/393a4c1e-cca3-4ff4-a31a-6d09d44dbb01-maas-api-service-ca\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:35.269163 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.269127    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"]
Sep 11 18:41:35.269365 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.269330    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk" podUID="c225c98b-9c37-4d3b-9548-3819ae93289c" containerName="payload-processing" containerID="cri-o://23d5dc1810743258683c6cc0c514778b5f0a806103b4fad6b872b7861a2288be" gracePeriod=30
Sep 11 18:41:35.557328 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.557305    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"
Sep 11 18:41:35.663670 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.663585    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps"]
Sep 11 18:41:35.665733 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.665708    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-869176-kh8ps"]
Sep 11 18:41:35.729837 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.729811    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c225c98b-9c37-4d3b-9548-3819ae93289c-plugins-config-volume\") pod \"c225c98b-9c37-4d3b-9548-3819ae93289c\" (UID: \"c225c98b-9c37-4d3b-9548-3819ae93289c\") "
Sep 11 18:41:35.729948 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.729856    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkfqz\" (UniqueName: \"kubernetes.io/projected/c225c98b-9c37-4d3b-9548-3819ae93289c-kube-api-access-vkfqz\") pod \"c225c98b-9c37-4d3b-9548-3819ae93289c\" (UID: \"c225c98b-9c37-4d3b-9548-3819ae93289c\") "
Sep 11 18:41:35.730196 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.730176    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c225c98b-9c37-4d3b-9548-3819ae93289c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c225c98b-9c37-4d3b-9548-3819ae93289c" (UID: "c225c98b-9c37-4d3b-9548-3819ae93289c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:41:35.730297 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.730285    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c225c98b-9c37-4d3b-9548-3819ae93289c-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:35.731998 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.731974    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c225c98b-9c37-4d3b-9548-3819ae93289c-kube-api-access-vkfqz" (OuterVolumeSpecName: "kube-api-access-vkfqz") pod "c225c98b-9c37-4d3b-9548-3819ae93289c" (UID: "c225c98b-9c37-4d3b-9548-3819ae93289c"). InnerVolumeSpecName "kube-api-access-vkfqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:35.765277 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.765250    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d3ff081-928d-40a8-b3aa-56160cc780ee" path="/var/lib/kubelet/pods/7d3ff081-928d-40a8-b3aa-56160cc780ee/volumes"
Sep 11 18:41:35.828560 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.828529    2581 generic.go:356] "Generic (PLEG): container finished" podID="c225c98b-9c37-4d3b-9548-3819ae93289c" containerID="23d5dc1810743258683c6cc0c514778b5f0a806103b4fad6b872b7861a2288be" exitCode=0
Sep 11 18:41:35.828650 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.828572    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"
Sep 11 18:41:35.828650 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.828616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk" event={"ID":"c225c98b-9c37-4d3b-9548-3819ae93289c","Type":"ContainerDied","Data":"23d5dc1810743258683c6cc0c514778b5f0a806103b4fad6b872b7861a2288be"}
Sep 11 18:41:35.828743 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.828664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk" event={"ID":"c225c98b-9c37-4d3b-9548-3819ae93289c","Type":"ContainerDied","Data":"26389c483aaa4642b5dbfe092ea2dc48c13f65c5a55b8733285e3aab67ddd8e5"}
Sep 11 18:41:35.828743 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.828686    2581 scope.go:117] "RemoveContainer" containerID="23d5dc1810743258683c6cc0c514778b5f0a806103b4fad6b872b7861a2288be"
Sep 11 18:41:35.830071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.830051    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk"
Sep 11 18:41:35.830346 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.830098    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk" event={"ID":"393a4c1e-cca3-4ff4-a31a-6d09d44dbb01","Type":"ContainerDied","Data":"c82b815419d1e36fbd961ec8faa7c56c6885c74bfce1da8aa0e5de54c4203a17"}
Sep 11 18:41:35.830346 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.830141    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c82b815419d1e36fbd961ec8faa7c56c6885c74bfce1da8aa0e5de54c4203a17"
Sep 11 18:41:35.830899 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.830874    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkfqz\" (UniqueName: \"kubernetes.io/projected/c225c98b-9c37-4d3b-9548-3819ae93289c-kube-api-access-vkfqz\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:35.838140 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.838120    2581 scope.go:117] "RemoveContainer" containerID="23d5dc1810743258683c6cc0c514778b5f0a806103b4fad6b872b7861a2288be"
Sep 11 18:41:35.838393 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:41:35.838376    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d5dc1810743258683c6cc0c514778b5f0a806103b4fad6b872b7861a2288be\": container with ID starting with 23d5dc1810743258683c6cc0c514778b5f0a806103b4fad6b872b7861a2288be not found: ID does not exist" containerID="23d5dc1810743258683c6cc0c514778b5f0a806103b4fad6b872b7861a2288be"
Sep 11 18:41:35.838446 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.838401    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d5dc1810743258683c6cc0c514778b5f0a806103b4fad6b872b7861a2288be"} err="failed to get container status \"23d5dc1810743258683c6cc0c514778b5f0a806103b4fad6b872b7861a2288be\": rpc error: code = NotFound desc = could not find container \"23d5dc1810743258683c6cc0c514778b5f0a806103b4fad6b872b7861a2288be\": container with ID starting with 23d5dc1810743258683c6cc0c514778b5f0a806103b4fad6b872b7861a2288be not found: ID does not exist"
Sep 11 18:41:35.846101 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.846081    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"]
Sep 11 18:41:35.850372 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:35.850353    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-869176-5d75f88975-qmrwk"]
Sep 11 18:41:36.841218 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:36.841183    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"]
Sep 11 18:41:36.858446 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:36.858413    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"]
Sep 11 18:41:36.870293 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:36.870267    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"]
Sep 11 18:41:37.764807 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:37.764776    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c225c98b-9c37-4d3b-9548-3819ae93289c" path="/var/lib/kubelet/pods/c225c98b-9c37-4d3b-9548-3819ae93289c/volumes"
Sep 11 18:41:43.685970 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:43.685936    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn"]
Sep 11 18:41:43.686382 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:43.686370    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393a4c1e-cca3-4ff4-a31a-6d09d44dbb01" containerName="revoke-keys"
Sep 11 18:41:43.686426 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:43.686384    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="393a4c1e-cca3-4ff4-a31a-6d09d44dbb01" containerName="revoke-keys"
Sep 11 18:41:43.686426 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:43.686402    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c225c98b-9c37-4d3b-9548-3819ae93289c" containerName="payload-processing"
Sep 11 18:41:43.686426 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:43.686407    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c225c98b-9c37-4d3b-9548-3819ae93289c" containerName="payload-processing"
Sep 11 18:41:43.686571 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:43.686489    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="393a4c1e-cca3-4ff4-a31a-6d09d44dbb01" containerName="revoke-keys"
Sep 11 18:41:43.686571 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:43.686498    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c225c98b-9c37-4d3b-9548-3819ae93289c" containerName="payload-processing"
Sep 11 18:41:43.691100 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:43.691081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn"
Sep 11 18:41:43.693173 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:43.693154    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\""
Sep 11 18:41:43.693288 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:43.693182    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:41:43.697593 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:43.697570    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn"]
Sep 11 18:41:43.806577 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:43.806545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d446d336-4c3f-407c-9ad1-7d9faf0064e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn\" (UID: \"d446d336-4c3f-407c-9ad1-7d9faf0064e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn"
Sep 11 18:41:43.907339 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:43.907309    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d446d336-4c3f-407c-9ad1-7d9faf0064e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn\" (UID: \"d446d336-4c3f-407c-9ad1-7d9faf0064e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn"
Sep 11 18:41:43.907949 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:43.907929    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d446d336-4c3f-407c-9ad1-7d9faf0064e7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn\" (UID: \"d446d336-4c3f-407c-9ad1-7d9faf0064e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn"
Sep 11 18:41:44.004379 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:44.004313    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn"
Sep 11 18:41:44.277374 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:44.277346    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn"]
Sep 11 18:41:44.279035 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:41:44.279008    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd446d336_4c3f_407c_9ad1_7d9faf0064e7.slice/crio-bc7a520a6da4f8f6f03177d05be63a78f7e3b719365ff7843651556b2521e68e WatchSource:0}: Error finding container bc7a520a6da4f8f6f03177d05be63a78f7e3b719365ff7843651556b2521e68e: Status 404 returned error can't find the container with id bc7a520a6da4f8f6f03177d05be63a78f7e3b719365ff7843651556b2521e68e
Sep 11 18:41:44.883261 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:44.883230    2581 generic.go:356] "Generic (PLEG): container finished" podID="d446d336-4c3f-407c-9ad1-7d9faf0064e7" containerID="aeb060e746fc7d349cddde5660630b9a50f4b9c020285e56611a05bd78ea82de" exitCode=0
Sep 11 18:41:44.883563 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:44.883307    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn" event={"ID":"d446d336-4c3f-407c-9ad1-7d9faf0064e7","Type":"ContainerDied","Data":"aeb060e746fc7d349cddde5660630b9a50f4b9c020285e56611a05bd78ea82de"}
Sep 11 18:41:44.883563 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:44.883357    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn" event={"ID":"d446d336-4c3f-407c-9ad1-7d9faf0064e7","Type":"ContainerStarted","Data":"bc7a520a6da4f8f6f03177d05be63a78f7e3b719365ff7843651556b2521e68e"}
Sep 11 18:41:45.288928 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.288894    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"]
Sep 11 18:41:45.289178 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.289147    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp" podUID="bebaebd3-07d3-45d4-b85e-364a006910b2" containerName="payload-processing" containerID="cri-o://ff629c35a750e74362054349a8eb0e75494e5182e0266f919bb4806c90a6199c" gracePeriod=30
Sep 11 18:41:45.298327 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.298302    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"]
Sep 11 18:41:45.298548 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.298526    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl" podUID="12b432db-84df-4c1c-b999-a20494bb1663" containerName="payload-pre-processing" containerID="cri-o://8b507b233cdb0f5141e8e4b5d9929e37bd9d87f2a4c9b24e33f1b64cee74da1f" gracePeriod=30
Sep 11 18:41:45.501274 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.501236    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk"]
Sep 11 18:41:45.504833 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.504799    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-efa8b9-jjdfk"]
Sep 11 18:41:45.575075 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.575050    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"
Sep 11 18:41:45.601262 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.601239    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"
Sep 11 18:41:45.625169 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.625133    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebaebd3-07d3-45d4-b85e-364a006910b2-plugins-config-volume\") pod \"bebaebd3-07d3-45d4-b85e-364a006910b2\" (UID: \"bebaebd3-07d3-45d4-b85e-364a006910b2\") "
Sep 11 18:41:45.625169 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.625168    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b432db-84df-4c1c-b999-a20494bb1663-plugins-config-volume\") pod \"12b432db-84df-4c1c-b999-a20494bb1663\" (UID: \"12b432db-84df-4c1c-b999-a20494bb1663\") "
Sep 11 18:41:45.625372 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.625267    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kf2x2\" (UniqueName: \"kubernetes.io/projected/bebaebd3-07d3-45d4-b85e-364a006910b2-kube-api-access-kf2x2\") pod \"bebaebd3-07d3-45d4-b85e-364a006910b2\" (UID: \"bebaebd3-07d3-45d4-b85e-364a006910b2\") "
Sep 11 18:41:45.625372 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.625295    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6q2b\" (UniqueName: \"kubernetes.io/projected/12b432db-84df-4c1c-b999-a20494bb1663-kube-api-access-w6q2b\") pod \"12b432db-84df-4c1c-b999-a20494bb1663\" (UID: \"12b432db-84df-4c1c-b999-a20494bb1663\") "
Sep 11 18:41:45.625524 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.625499    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bebaebd3-07d3-45d4-b85e-364a006910b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bebaebd3-07d3-45d4-b85e-364a006910b2" (UID: "bebaebd3-07d3-45d4-b85e-364a006910b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:41:45.625596 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.625537    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b432db-84df-4c1c-b999-a20494bb1663-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "12b432db-84df-4c1c-b999-a20494bb1663" (UID: "12b432db-84df-4c1c-b999-a20494bb1663"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:41:45.625658 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.625630    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bebaebd3-07d3-45d4-b85e-364a006910b2-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:45.625658 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.625650    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12b432db-84df-4c1c-b999-a20494bb1663-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:45.627395 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.627370    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bebaebd3-07d3-45d4-b85e-364a006910b2-kube-api-access-kf2x2" (OuterVolumeSpecName: "kube-api-access-kf2x2") pod "bebaebd3-07d3-45d4-b85e-364a006910b2" (UID: "bebaebd3-07d3-45d4-b85e-364a006910b2"). InnerVolumeSpecName "kube-api-access-kf2x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:45.627515 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.627437    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b432db-84df-4c1c-b999-a20494bb1663-kube-api-access-w6q2b" (OuterVolumeSpecName: "kube-api-access-w6q2b") pod "12b432db-84df-4c1c-b999-a20494bb1663" (UID: "12b432db-84df-4c1c-b999-a20494bb1663"). InnerVolumeSpecName "kube-api-access-w6q2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:45.726624 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.726599    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf2x2\" (UniqueName: \"kubernetes.io/projected/bebaebd3-07d3-45d4-b85e-364a006910b2-kube-api-access-kf2x2\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:45.726624 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.726621    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6q2b\" (UniqueName: \"kubernetes.io/projected/12b432db-84df-4c1c-b999-a20494bb1663-kube-api-access-w6q2b\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:45.764437 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.764413    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="393a4c1e-cca3-4ff4-a31a-6d09d44dbb01" path="/var/lib/kubelet/pods/393a4c1e-cca3-4ff4-a31a-6d09d44dbb01/volumes"
Sep 11 18:41:45.889887 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.889855    2581 generic.go:356] "Generic (PLEG): container finished" podID="bebaebd3-07d3-45d4-b85e-364a006910b2" containerID="ff629c35a750e74362054349a8eb0e75494e5182e0266f919bb4806c90a6199c" exitCode=0
Sep 11 18:41:45.890296 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.889899    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"
Sep 11 18:41:45.890296 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.889943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp" event={"ID":"bebaebd3-07d3-45d4-b85e-364a006910b2","Type":"ContainerDied","Data":"ff629c35a750e74362054349a8eb0e75494e5182e0266f919bb4806c90a6199c"}
Sep 11 18:41:45.890296 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.890000    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp" event={"ID":"bebaebd3-07d3-45d4-b85e-364a006910b2","Type":"ContainerDied","Data":"eaac3265115a94b7adfe7e752396dae25a7a66d27888e29939e81c3f92155903"}
Sep 11 18:41:45.890296 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.890029    2581 scope.go:117] "RemoveContainer" containerID="ff629c35a750e74362054349a8eb0e75494e5182e0266f919bb4806c90a6199c"
Sep 11 18:41:45.891416 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.891394    2581 generic.go:356] "Generic (PLEG): container finished" podID="12b432db-84df-4c1c-b999-a20494bb1663" containerID="8b507b233cdb0f5141e8e4b5d9929e37bd9d87f2a4c9b24e33f1b64cee74da1f" exitCode=0
Sep 11 18:41:45.891557 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.891434    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"
Sep 11 18:41:45.891557 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.891465    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl" event={"ID":"12b432db-84df-4c1c-b999-a20494bb1663","Type":"ContainerDied","Data":"8b507b233cdb0f5141e8e4b5d9929e37bd9d87f2a4c9b24e33f1b64cee74da1f"}
Sep 11 18:41:45.891557 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.891526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl" event={"ID":"12b432db-84df-4c1c-b999-a20494bb1663","Type":"ContainerDied","Data":"6a7d79b0c0e7a908c05179a6121839a28ec400c59cc4a688a149ff2b28fec14b"}
Sep 11 18:41:45.900996 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.900969    2581 scope.go:117] "RemoveContainer" containerID="ff629c35a750e74362054349a8eb0e75494e5182e0266f919bb4806c90a6199c"
Sep 11 18:41:45.901256 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:41:45.901239    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff629c35a750e74362054349a8eb0e75494e5182e0266f919bb4806c90a6199c\": container with ID starting with ff629c35a750e74362054349a8eb0e75494e5182e0266f919bb4806c90a6199c not found: ID does not exist" containerID="ff629c35a750e74362054349a8eb0e75494e5182e0266f919bb4806c90a6199c"
Sep 11 18:41:45.901322 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.901270    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff629c35a750e74362054349a8eb0e75494e5182e0266f919bb4806c90a6199c"} err="failed to get container status \"ff629c35a750e74362054349a8eb0e75494e5182e0266f919bb4806c90a6199c\": rpc error: code = NotFound desc = could not find container \"ff629c35a750e74362054349a8eb0e75494e5182e0266f919bb4806c90a6199c\": container with ID starting with ff629c35a750e74362054349a8eb0e75494e5182e0266f919bb4806c90a6199c not found: ID does not exist"
Sep 11 18:41:45.901322 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.901286    2581 scope.go:117] "RemoveContainer" containerID="8b507b233cdb0f5141e8e4b5d9929e37bd9d87f2a4c9b24e33f1b64cee74da1f"
Sep 11 18:41:45.911645 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.911620    2581 scope.go:117] "RemoveContainer" containerID="8b507b233cdb0f5141e8e4b5d9929e37bd9d87f2a4c9b24e33f1b64cee74da1f"
Sep 11 18:41:45.911933 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:41:45.911912    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b507b233cdb0f5141e8e4b5d9929e37bd9d87f2a4c9b24e33f1b64cee74da1f\": container with ID starting with 8b507b233cdb0f5141e8e4b5d9929e37bd9d87f2a4c9b24e33f1b64cee74da1f not found: ID does not exist" containerID="8b507b233cdb0f5141e8e4b5d9929e37bd9d87f2a4c9b24e33f1b64cee74da1f"
Sep 11 18:41:45.912010 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.911942    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b507b233cdb0f5141e8e4b5d9929e37bd9d87f2a4c9b24e33f1b64cee74da1f"} err="failed to get container status \"8b507b233cdb0f5141e8e4b5d9929e37bd9d87f2a4c9b24e33f1b64cee74da1f\": rpc error: code = NotFound desc = could not find container \"8b507b233cdb0f5141e8e4b5d9929e37bd9d87f2a4c9b24e33f1b64cee74da1f\": container with ID starting with 8b507b233cdb0f5141e8e4b5d9929e37bd9d87f2a4c9b24e33f1b64cee74da1f not found: ID does not exist"
Sep 11 18:41:45.912054 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.912005    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"]
Sep 11 18:41:45.916188 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.916169    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-efa8b9-74798786cc-jf8wp"]
Sep 11 18:41:45.930596 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.930576    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"]
Sep 11 18:41:45.945799 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:45.945776    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-efa8b9-658f964b5d-kcknl"]
Sep 11 18:41:46.012116 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:46.012087    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn"
Sep 11 18:41:46.131206 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:46.131177    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d446d336-4c3f-407c-9ad1-7d9faf0064e7-maas-api-service-ca\") pod \"d446d336-4c3f-407c-9ad1-7d9faf0064e7\" (UID: \"d446d336-4c3f-407c-9ad1-7d9faf0064e7\") "
Sep 11 18:41:46.131565 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:46.131542    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d446d336-4c3f-407c-9ad1-7d9faf0064e7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d446d336-4c3f-407c-9ad1-7d9faf0064e7" (UID: "d446d336-4c3f-407c-9ad1-7d9faf0064e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:41:46.232248 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:46.232173    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d446d336-4c3f-407c-9ad1-7d9faf0064e7-maas-api-service-ca\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:46.898987 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:46.898960    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn"
Sep 11 18:41:46.899441 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:46.899001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn" event={"ID":"d446d336-4c3f-407c-9ad1-7d9faf0064e7","Type":"ContainerDied","Data":"bc7a520a6da4f8f6f03177d05be63a78f7e3b719365ff7843651556b2521e68e"}
Sep 11 18:41:46.899441 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:46.899047    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc7a520a6da4f8f6f03177d05be63a78f7e3b719365ff7843651556b2521e68e"
Sep 11 18:41:47.764300 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:47.764263    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12b432db-84df-4c1c-b999-a20494bb1663" path="/var/lib/kubelet/pods/12b432db-84df-4c1c-b999-a20494bb1663/volumes"
Sep 11 18:41:47.764630 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:47.764614    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bebaebd3-07d3-45d4-b85e-364a006910b2" path="/var/lib/kubelet/pods/bebaebd3-07d3-45d4-b85e-364a006910b2/volumes"
Sep 11 18:41:51.860885 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:51.860837    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh" podUID="31321600-870d-4814-ab30-121514d58e5d" containerName="main" containerID="cri-o://b08ea24ebe7be2de0c4fe36a0f0bdcc3d7a4da3a4c8e46961f1d87f9778362b8" gracePeriod=45
Sep 11 18:41:51.877264 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:51.877229    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z" podUID="c052afe1-73fc-4a08-9299-aa8dea7b8d5e" containerName="main" containerID="cri-o://64c234f114780acc98ec1212984f2ed6057880be1004a1fcca9152ce0545bea0" gracePeriod=45
Sep 11 18:41:51.888946 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:51.888910    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6" podUID="037f2be3-9585-409a-adbc-7ca76eb13efc" containerName="main" containerID="cri-o://8141f2b74570ef5533246946238064db73a314d1811b87c9a1cecdf770065105" gracePeriod=45
Sep 11 18:41:52.206555 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.206460    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:41:52.209319 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.209297    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:41:52.212895 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.212880    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:41:52.288401 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.288369    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-tls-certs\") pod \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") "
Sep 11 18:41:52.288612 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.288411    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-dshm\") pod \"31321600-870d-4814-ab30-121514d58e5d\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") "
Sep 11 18:41:52.288612 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.288447    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-dshm\") pod \"037f2be3-9585-409a-adbc-7ca76eb13efc\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") "
Sep 11 18:41:52.288612 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.288558    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-tmp-dir\") pod \"31321600-870d-4814-ab30-121514d58e5d\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") "
Sep 11 18:41:52.288806 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.288620    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4drtl\" (UniqueName: \"kubernetes.io/projected/037f2be3-9585-409a-adbc-7ca76eb13efc-kube-api-access-4drtl\") pod \"037f2be3-9585-409a-adbc-7ca76eb13efc\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") "
Sep 11 18:41:52.288806 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.288645    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-dshm\") pod \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") "
Sep 11 18:41:52.288806 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.288677    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-home\") pod \"31321600-870d-4814-ab30-121514d58e5d\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") "
Sep 11 18:41:52.288806 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.288704    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/037f2be3-9585-409a-adbc-7ca76eb13efc-tls-certs\") pod \"037f2be3-9585-409a-adbc-7ca76eb13efc\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") "
Sep 11 18:41:52.288806 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.288739    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-home\") pod \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") "
Sep 11 18:41:52.288806 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.288760    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-model-cache\") pod \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") "
Sep 11 18:41:52.289123 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.288814    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-tmp-dir\") pod \"037f2be3-9585-409a-adbc-7ca76eb13efc\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") "
Sep 11 18:41:52.289123 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.288857    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-home\") pod \"037f2be3-9585-409a-adbc-7ca76eb13efc\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") "
Sep 11 18:41:52.289123 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.288913    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67xmj\" (UniqueName: \"kubernetes.io/projected/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-kube-api-access-67xmj\") pod \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") "
Sep 11 18:41:52.289123 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.288936    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31321600-870d-4814-ab30-121514d58e5d-tls-certs\") pod \"31321600-870d-4814-ab30-121514d58e5d\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") "
Sep 11 18:41:52.289123 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.288979    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-model-cache\") pod \"31321600-870d-4814-ab30-121514d58e5d\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") "
Sep 11 18:41:52.289123 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.289015    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pnp6\" (UniqueName: \"kubernetes.io/projected/31321600-870d-4814-ab30-121514d58e5d-kube-api-access-2pnp6\") pod \"31321600-870d-4814-ab30-121514d58e5d\" (UID: \"31321600-870d-4814-ab30-121514d58e5d\") "
Sep 11 18:41:52.289123 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.289078    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-tmp-dir\") pod \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\" (UID: \"c052afe1-73fc-4a08-9299-aa8dea7b8d5e\") "
Sep 11 18:41:52.289123 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.289110    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-model-cache\") pod \"037f2be3-9585-409a-adbc-7ca76eb13efc\" (UID: \"037f2be3-9585-409a-adbc-7ca76eb13efc\") "
Sep 11 18:41:52.289931 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.289903    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-model-cache" (OuterVolumeSpecName: "model-cache") pod "037f2be3-9585-409a-adbc-7ca76eb13efc" (UID: "037f2be3-9585-409a-adbc-7ca76eb13efc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.290325 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.290090    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-home" (OuterVolumeSpecName: "home") pod "31321600-870d-4814-ab30-121514d58e5d" (UID: "31321600-870d-4814-ab30-121514d58e5d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.290702 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.290669    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-model-cache" (OuterVolumeSpecName: "model-cache") pod "31321600-870d-4814-ab30-121514d58e5d" (UID: "31321600-870d-4814-ab30-121514d58e5d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.291146 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.290978    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31321600-870d-4814-ab30-121514d58e5d" (UID: "31321600-870d-4814-ab30-121514d58e5d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.291461 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.291282    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "c052afe1-73fc-4a08-9299-aa8dea7b8d5e" (UID: "c052afe1-73fc-4a08-9299-aa8dea7b8d5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.291461 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.291371    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c052afe1-73fc-4a08-9299-aa8dea7b8d5e" (UID: "c052afe1-73fc-4a08-9299-aa8dea7b8d5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:52.291461 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.291383    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-dshm" (OuterVolumeSpecName: "dshm") pod "31321600-870d-4814-ab30-121514d58e5d" (UID: "31321600-870d-4814-ab30-121514d58e5d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.291461 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.291428    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-dshm" (OuterVolumeSpecName: "dshm") pod "037f2be3-9585-409a-adbc-7ca76eb13efc" (UID: "037f2be3-9585-409a-adbc-7ca76eb13efc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.291743 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.291674    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-home" (OuterVolumeSpecName: "home") pod "037f2be3-9585-409a-adbc-7ca76eb13efc" (UID: "037f2be3-9585-409a-adbc-7ca76eb13efc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.291886 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.291806    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "037f2be3-9585-409a-adbc-7ca76eb13efc" (UID: "037f2be3-9585-409a-adbc-7ca76eb13efc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.291886 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.291842    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-home" (OuterVolumeSpecName: "home") pod "c052afe1-73fc-4a08-9299-aa8dea7b8d5e" (UID: "c052afe1-73fc-4a08-9299-aa8dea7b8d5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.292008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.291885    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c052afe1-73fc-4a08-9299-aa8dea7b8d5e" (UID: "c052afe1-73fc-4a08-9299-aa8dea7b8d5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.292193 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.292117    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-dshm" (OuterVolumeSpecName: "dshm") pod "c052afe1-73fc-4a08-9299-aa8dea7b8d5e" (UID: "c052afe1-73fc-4a08-9299-aa8dea7b8d5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:41:52.292939 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.292906    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31321600-870d-4814-ab30-121514d58e5d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31321600-870d-4814-ab30-121514d58e5d" (UID: "31321600-870d-4814-ab30-121514d58e5d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:52.293389 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.293364    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/037f2be3-9585-409a-adbc-7ca76eb13efc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "037f2be3-9585-409a-adbc-7ca76eb13efc" (UID: "037f2be3-9585-409a-adbc-7ca76eb13efc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:41:52.293886 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.293860    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31321600-870d-4814-ab30-121514d58e5d-kube-api-access-2pnp6" (OuterVolumeSpecName: "kube-api-access-2pnp6") pod "31321600-870d-4814-ab30-121514d58e5d" (UID: "31321600-870d-4814-ab30-121514d58e5d"). InnerVolumeSpecName "kube-api-access-2pnp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:52.294169 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.294133    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-kube-api-access-67xmj" (OuterVolumeSpecName: "kube-api-access-67xmj") pod "c052afe1-73fc-4a08-9299-aa8dea7b8d5e" (UID: "c052afe1-73fc-4a08-9299-aa8dea7b8d5e"). InnerVolumeSpecName "kube-api-access-67xmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:52.294350 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.294330    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037f2be3-9585-409a-adbc-7ca76eb13efc-kube-api-access-4drtl" (OuterVolumeSpecName: "kube-api-access-4drtl") pod "037f2be3-9585-409a-adbc-7ca76eb13efc" (UID: "037f2be3-9585-409a-adbc-7ca76eb13efc"). InnerVolumeSpecName "kube-api-access-4drtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:52.390511 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390455    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390511 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390511    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390521    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390531    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4drtl\" (UniqueName: \"kubernetes.io/projected/037f2be3-9585-409a-adbc-7ca76eb13efc-kube-api-access-4drtl\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390539    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390546    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390553    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/037f2be3-9585-409a-adbc-7ca76eb13efc-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390559    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390566    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390573    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390580    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390587    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67xmj\" (UniqueName: \"kubernetes.io/projected/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-kube-api-access-67xmj\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390594    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31321600-870d-4814-ab30-121514d58e5d-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390601    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31321600-870d-4814-ab30-121514d58e5d-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390608    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pnp6\" (UniqueName: \"kubernetes.io/projected/31321600-870d-4814-ab30-121514d58e5d-kube-api-access-2pnp6\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390615    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390623    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/037f2be3-9585-409a-adbc-7ca76eb13efc-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.390648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.390629    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c052afe1-73fc-4a08-9299-aa8dea7b8d5e-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:52.937011 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.936980    2581 generic.go:356] "Generic (PLEG): container finished" podID="31321600-870d-4814-ab30-121514d58e5d" containerID="b08ea24ebe7be2de0c4fe36a0f0bdcc3d7a4da3a4c8e46961f1d87f9778362b8" exitCode=0
Sep 11 18:41:52.937392 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.937021    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"
Sep 11 18:41:52.937392 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.937063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh" event={"ID":"31321600-870d-4814-ab30-121514d58e5d","Type":"ContainerDied","Data":"b08ea24ebe7be2de0c4fe36a0f0bdcc3d7a4da3a4c8e46961f1d87f9778362b8"}
Sep 11 18:41:52.937392 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.937096    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh" event={"ID":"31321600-870d-4814-ab30-121514d58e5d","Type":"ContainerDied","Data":"e13bf0c719e36e11395dcc92fdbd9417852921dd2bb018a374cce759cfe5e481"}
Sep 11 18:41:52.937392 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.937114    2581 scope.go:117] "RemoveContainer" containerID="b08ea24ebe7be2de0c4fe36a0f0bdcc3d7a4da3a4c8e46961f1d87f9778362b8"
Sep 11 18:41:52.938660 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.938629    2581 generic.go:356] "Generic (PLEG): container finished" podID="037f2be3-9585-409a-adbc-7ca76eb13efc" containerID="8141f2b74570ef5533246946238064db73a314d1811b87c9a1cecdf770065105" exitCode=0
Sep 11 18:41:52.938789 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.938665    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"
Sep 11 18:41:52.938789 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.938718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6" event={"ID":"037f2be3-9585-409a-adbc-7ca76eb13efc","Type":"ContainerDied","Data":"8141f2b74570ef5533246946238064db73a314d1811b87c9a1cecdf770065105"}
Sep 11 18:41:52.938789 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.938756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6" event={"ID":"037f2be3-9585-409a-adbc-7ca76eb13efc","Type":"ContainerDied","Data":"f5e74bd57516c121c6a97ae69eb22202903f9f43ebc54d59de2e53e15d4c0e1b"}
Sep 11 18:41:52.940272 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.940239    2581 generic.go:356] "Generic (PLEG): container finished" podID="c052afe1-73fc-4a08-9299-aa8dea7b8d5e" containerID="64c234f114780acc98ec1212984f2ed6057880be1004a1fcca9152ce0545bea0" exitCode=0
Sep 11 18:41:52.940395 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.940278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z" event={"ID":"c052afe1-73fc-4a08-9299-aa8dea7b8d5e","Type":"ContainerDied","Data":"64c234f114780acc98ec1212984f2ed6057880be1004a1fcca9152ce0545bea0"}
Sep 11 18:41:52.940395 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.940284    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"
Sep 11 18:41:52.940395 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.940304    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z" event={"ID":"c052afe1-73fc-4a08-9299-aa8dea7b8d5e","Type":"ContainerDied","Data":"707a1b85018758219706c14c8f2100a155c1fdb46ae673c74eb538de1b5321dd"}
Sep 11 18:41:52.947600 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.947528    2581 scope.go:117] "RemoveContainer" containerID="b08ea24ebe7be2de0c4fe36a0f0bdcc3d7a4da3a4c8e46961f1d87f9778362b8"
Sep 11 18:41:52.947962 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:41:52.947810    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08ea24ebe7be2de0c4fe36a0f0bdcc3d7a4da3a4c8e46961f1d87f9778362b8\": container with ID starting with b08ea24ebe7be2de0c4fe36a0f0bdcc3d7a4da3a4c8e46961f1d87f9778362b8 not found: ID does not exist" containerID="b08ea24ebe7be2de0c4fe36a0f0bdcc3d7a4da3a4c8e46961f1d87f9778362b8"
Sep 11 18:41:52.947962 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.947838    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08ea24ebe7be2de0c4fe36a0f0bdcc3d7a4da3a4c8e46961f1d87f9778362b8"} err="failed to get container status \"b08ea24ebe7be2de0c4fe36a0f0bdcc3d7a4da3a4c8e46961f1d87f9778362b8\": rpc error: code = NotFound desc = could not find container \"b08ea24ebe7be2de0c4fe36a0f0bdcc3d7a4da3a4c8e46961f1d87f9778362b8\": container with ID starting with b08ea24ebe7be2de0c4fe36a0f0bdcc3d7a4da3a4c8e46961f1d87f9778362b8 not found: ID does not exist"
Sep 11 18:41:52.947962 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.947861    2581 scope.go:117] "RemoveContainer" containerID="8141f2b74570ef5533246946238064db73a314d1811b87c9a1cecdf770065105"
Sep 11 18:41:52.956076 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.956058    2581 scope.go:117] "RemoveContainer" containerID="8141f2b74570ef5533246946238064db73a314d1811b87c9a1cecdf770065105"
Sep 11 18:41:52.956318 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:41:52.956304    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8141f2b74570ef5533246946238064db73a314d1811b87c9a1cecdf770065105\": container with ID starting with 8141f2b74570ef5533246946238064db73a314d1811b87c9a1cecdf770065105 not found: ID does not exist" containerID="8141f2b74570ef5533246946238064db73a314d1811b87c9a1cecdf770065105"
Sep 11 18:41:52.956379 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.956323    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8141f2b74570ef5533246946238064db73a314d1811b87c9a1cecdf770065105"} err="failed to get container status \"8141f2b74570ef5533246946238064db73a314d1811b87c9a1cecdf770065105\": rpc error: code = NotFound desc = could not find container \"8141f2b74570ef5533246946238064db73a314d1811b87c9a1cecdf770065105\": container with ID starting with 8141f2b74570ef5533246946238064db73a314d1811b87c9a1cecdf770065105 not found: ID does not exist"
Sep 11 18:41:52.956379 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.956345    2581 scope.go:117] "RemoveContainer" containerID="64c234f114780acc98ec1212984f2ed6057880be1004a1fcca9152ce0545bea0"
Sep 11 18:41:52.964033 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.964018    2581 scope.go:117] "RemoveContainer" containerID="64c234f114780acc98ec1212984f2ed6057880be1004a1fcca9152ce0545bea0"
Sep 11 18:41:52.964271 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:41:52.964253    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c234f114780acc98ec1212984f2ed6057880be1004a1fcca9152ce0545bea0\": container with ID starting with 64c234f114780acc98ec1212984f2ed6057880be1004a1fcca9152ce0545bea0 not found: ID does not exist" containerID="64c234f114780acc98ec1212984f2ed6057880be1004a1fcca9152ce0545bea0"
Sep 11 18:41:52.964317 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.964277    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c234f114780acc98ec1212984f2ed6057880be1004a1fcca9152ce0545bea0"} err="failed to get container status \"64c234f114780acc98ec1212984f2ed6057880be1004a1fcca9152ce0545bea0\": rpc error: code = NotFound desc = could not find container \"64c234f114780acc98ec1212984f2ed6057880be1004a1fcca9152ce0545bea0\": container with ID starting with 64c234f114780acc98ec1212984f2ed6057880be1004a1fcca9152ce0545bea0 not found: ID does not exist"
Sep 11 18:41:52.971997 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.971980    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"]
Sep 11 18:41:52.979268 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.979249    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/simulator-w0-efa8b9-kserve-5d5577f696-zc4g6"]
Sep 11 18:41:52.992138 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:52.992116    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"]
Sep 11 18:41:53.002932 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:53.002913    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/distinct-w0-efa8b9-kserve-56c66d4c8d-695xh"]
Sep 11 18:41:53.015796 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:53.015778    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"]
Sep 11 18:41:53.020454 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:53.020432    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-efa8b9/premium-w0-efa8b9-kserve-684d99fdfc-sft2z"]
Sep 11 18:41:53.770488 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:53.770455    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="037f2be3-9585-409a-adbc-7ca76eb13efc" path="/var/lib/kubelet/pods/037f2be3-9585-409a-adbc-7ca76eb13efc/volumes"
Sep 11 18:41:53.770852 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:53.770839    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31321600-870d-4814-ab30-121514d58e5d" path="/var/lib/kubelet/pods/31321600-870d-4814-ab30-121514d58e5d/volumes"
Sep 11 18:41:53.771171 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:53.771160    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c052afe1-73fc-4a08-9299-aa8dea7b8d5e" path="/var/lib/kubelet/pods/c052afe1-73fc-4a08-9299-aa8dea7b8d5e/volumes"
Sep 11 18:41:54.151221 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:54.151191    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:41:54.843700 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:54.843665    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"]
Sep 11 18:41:54.843915 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:54.843885    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76" podUID="6b1e464f-bf95-4b38-8112-f5003d21c234" containerName="payload-pre-processing" containerID="cri-o://6fa6fa2eb25488cfbd53a3d9a996ab9b47fa0a3030b7112f5329bdc612ccdcac" gracePeriod=30
Sep 11 18:41:55.087191 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.087163    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn"]
Sep 11 18:41:55.090350 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.090328    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-0ead89-57mwn"]
Sep 11 18:41:55.094196 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.094147    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"
Sep 11 18:41:55.215592 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.215561    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b1e464f-bf95-4b38-8112-f5003d21c234-plugins-config-volume\") pod \"6b1e464f-bf95-4b38-8112-f5003d21c234\" (UID: \"6b1e464f-bf95-4b38-8112-f5003d21c234\") "
Sep 11 18:41:55.215942 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.215670    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9swnv\" (UniqueName: \"kubernetes.io/projected/6b1e464f-bf95-4b38-8112-f5003d21c234-kube-api-access-9swnv\") pod \"6b1e464f-bf95-4b38-8112-f5003d21c234\" (UID: \"6b1e464f-bf95-4b38-8112-f5003d21c234\") "
Sep 11 18:41:55.215942 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.215897    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b1e464f-bf95-4b38-8112-f5003d21c234-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b1e464f-bf95-4b38-8112-f5003d21c234" (UID: "6b1e464f-bf95-4b38-8112-f5003d21c234"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:41:55.216015 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.215963    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b1e464f-bf95-4b38-8112-f5003d21c234-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:55.217956 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.217931    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1e464f-bf95-4b38-8112-f5003d21c234-kube-api-access-9swnv" (OuterVolumeSpecName: "kube-api-access-9swnv") pod "6b1e464f-bf95-4b38-8112-f5003d21c234" (UID: "6b1e464f-bf95-4b38-8112-f5003d21c234"). InnerVolumeSpecName "kube-api-access-9swnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:41:55.317408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.317378    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9swnv\" (UniqueName: \"kubernetes.io/projected/6b1e464f-bf95-4b38-8112-f5003d21c234-kube-api-access-9swnv\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:41:55.764149 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.764114    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d446d336-4c3f-407c-9ad1-7d9faf0064e7" path="/var/lib/kubelet/pods/d446d336-4c3f-407c-9ad1-7d9faf0064e7/volumes"
Sep 11 18:41:55.961692 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.961657    2581 generic.go:356] "Generic (PLEG): container finished" podID="6b1e464f-bf95-4b38-8112-f5003d21c234" containerID="6fa6fa2eb25488cfbd53a3d9a996ab9b47fa0a3030b7112f5329bdc612ccdcac" exitCode=0
Sep 11 18:41:55.961856 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.961704    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"
Sep 11 18:41:55.961856 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.961738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76" event={"ID":"6b1e464f-bf95-4b38-8112-f5003d21c234","Type":"ContainerDied","Data":"6fa6fa2eb25488cfbd53a3d9a996ab9b47fa0a3030b7112f5329bdc612ccdcac"}
Sep 11 18:41:55.961856 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.961796    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76" event={"ID":"6b1e464f-bf95-4b38-8112-f5003d21c234","Type":"ContainerDied","Data":"e3e95da149191a41b4a9d6ab2587ff83dc64655253789779a1d1688e8352dafb"}
Sep 11 18:41:55.961856 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.961822    2581 scope.go:117] "RemoveContainer" containerID="6fa6fa2eb25488cfbd53a3d9a996ab9b47fa0a3030b7112f5329bdc612ccdcac"
Sep 11 18:41:55.970706 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.970680    2581 scope.go:117] "RemoveContainer" containerID="6fa6fa2eb25488cfbd53a3d9a996ab9b47fa0a3030b7112f5329bdc612ccdcac"
Sep 11 18:41:55.970980 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:41:55.970959    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fa6fa2eb25488cfbd53a3d9a996ab9b47fa0a3030b7112f5329bdc612ccdcac\": container with ID starting with 6fa6fa2eb25488cfbd53a3d9a996ab9b47fa0a3030b7112f5329bdc612ccdcac not found: ID does not exist" containerID="6fa6fa2eb25488cfbd53a3d9a996ab9b47fa0a3030b7112f5329bdc612ccdcac"
Sep 11 18:41:55.971027 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.970988    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fa6fa2eb25488cfbd53a3d9a996ab9b47fa0a3030b7112f5329bdc612ccdcac"} err="failed to get container status \"6fa6fa2eb25488cfbd53a3d9a996ab9b47fa0a3030b7112f5329bdc612ccdcac\": rpc error: code = NotFound desc = could not find container \"6fa6fa2eb25488cfbd53a3d9a996ab9b47fa0a3030b7112f5329bdc612ccdcac\": container with ID starting with 6fa6fa2eb25488cfbd53a3d9a996ab9b47fa0a3030b7112f5329bdc612ccdcac not found: ID does not exist"
Sep 11 18:41:55.982000 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.981982    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"]
Sep 11 18:41:55.989738 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:55.989718    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-0ead89-f5b44c486-mtp76"]
Sep 11 18:41:57.763895 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:41:57.763852    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b1e464f-bf95-4b38-8112-f5003d21c234" path="/var/lib/kubelet/pods/6b1e464f-bf95-4b38-8112-f5003d21c234/volumes"
Sep 11 18:42:16.862010 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.861937    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg"]
Sep 11 18:42:16.862342 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862317    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="037f2be3-9585-409a-adbc-7ca76eb13efc" containerName="main"
Sep 11 18:42:16.862342 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862327    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="037f2be3-9585-409a-adbc-7ca76eb13efc" containerName="main"
Sep 11 18:42:16.862342 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862336    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c052afe1-73fc-4a08-9299-aa8dea7b8d5e" containerName="main"
Sep 11 18:42:16.862342 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862341    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c052afe1-73fc-4a08-9299-aa8dea7b8d5e" containerName="main"
Sep 11 18:42:16.862506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862346    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b1e464f-bf95-4b38-8112-f5003d21c234" containerName="payload-pre-processing"
Sep 11 18:42:16.862506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862351    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b1e464f-bf95-4b38-8112-f5003d21c234" containerName="payload-pre-processing"
Sep 11 18:42:16.862506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862359    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12b432db-84df-4c1c-b999-a20494bb1663" containerName="payload-pre-processing"
Sep 11 18:42:16.862506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862365    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="12b432db-84df-4c1c-b999-a20494bb1663" containerName="payload-pre-processing"
Sep 11 18:42:16.862506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862380    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31321600-870d-4814-ab30-121514d58e5d" containerName="main"
Sep 11 18:42:16.862506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862386    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="31321600-870d-4814-ab30-121514d58e5d" containerName="main"
Sep 11 18:42:16.862506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862392    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bebaebd3-07d3-45d4-b85e-364a006910b2" containerName="payload-processing"
Sep 11 18:42:16.862506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862398    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bebaebd3-07d3-45d4-b85e-364a006910b2" containerName="payload-processing"
Sep 11 18:42:16.862506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862405    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d446d336-4c3f-407c-9ad1-7d9faf0064e7" containerName="revoke-keys"
Sep 11 18:42:16.862506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862410    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d446d336-4c3f-407c-9ad1-7d9faf0064e7" containerName="revoke-keys"
Sep 11 18:42:16.862506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862495    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b1e464f-bf95-4b38-8112-f5003d21c234" containerName="payload-pre-processing"
Sep 11 18:42:16.862506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862502    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c052afe1-73fc-4a08-9299-aa8dea7b8d5e" containerName="main"
Sep 11 18:42:16.862506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862508    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d446d336-4c3f-407c-9ad1-7d9faf0064e7" containerName="revoke-keys"
Sep 11 18:42:16.862892 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862516    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="037f2be3-9585-409a-adbc-7ca76eb13efc" containerName="main"
Sep 11 18:42:16.862892 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862523    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bebaebd3-07d3-45d4-b85e-364a006910b2" containerName="payload-processing"
Sep 11 18:42:16.862892 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862528    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="12b432db-84df-4c1c-b999-a20494bb1663" containerName="payload-pre-processing"
Sep 11 18:42:16.862892 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.862534    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="31321600-870d-4814-ab30-121514d58e5d" containerName="main"
Sep 11 18:42:16.867039 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.867024    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg"
Sep 11 18:42:16.869099 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.869075    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\""
Sep 11 18:42:16.869099 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.869085    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:42:16.882902 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.882876    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg"]
Sep 11 18:42:16.917346 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:16.917318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50746007-2cbb-4739-8877-32dc05178493-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg\" (UID: \"50746007-2cbb-4739-8877-32dc05178493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg"
Sep 11 18:42:17.018678 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:17.018651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50746007-2cbb-4739-8877-32dc05178493-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg\" (UID: \"50746007-2cbb-4739-8877-32dc05178493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg"
Sep 11 18:42:17.019224 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:17.019198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50746007-2cbb-4739-8877-32dc05178493-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg\" (UID: \"50746007-2cbb-4739-8877-32dc05178493\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg"
Sep 11 18:42:17.176739 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:17.176712    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg"
Sep 11 18:42:17.455649 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:17.455615    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg"]
Sep 11 18:42:17.456820 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:42:17.456795    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50746007_2cbb_4739_8877_32dc05178493.slice/crio-2de1d5e886f9ac419bf3f7c0cad6c1dfc66ea2650f38aed4f1f8ea2e0f48c4a9 WatchSource:0}: Error finding container 2de1d5e886f9ac419bf3f7c0cad6c1dfc66ea2650f38aed4f1f8ea2e0f48c4a9: Status 404 returned error can't find the container with id 2de1d5e886f9ac419bf3f7c0cad6c1dfc66ea2650f38aed4f1f8ea2e0f48c4a9
Sep 11 18:42:18.082832 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:18.082755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg" event={"ID":"50746007-2cbb-4739-8877-32dc05178493","Type":"ContainerStarted","Data":"1943ed034338fe95406594fcd052586adb2fe8cd83d72b98dddb58331aeccba1"}
Sep 11 18:42:18.082832 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:18.082802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg" event={"ID":"50746007-2cbb-4739-8877-32dc05178493","Type":"ContainerStarted","Data":"2de1d5e886f9ac419bf3f7c0cad6c1dfc66ea2650f38aed4f1f8ea2e0f48c4a9"}
Sep 11 18:42:19.089460 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:19.089427    2581 generic.go:356] "Generic (PLEG): container finished" podID="50746007-2cbb-4739-8877-32dc05178493" containerID="1943ed034338fe95406594fcd052586adb2fe8cd83d72b98dddb58331aeccba1" exitCode=0
Sep 11 18:42:19.089849 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:19.089472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg" event={"ID":"50746007-2cbb-4739-8877-32dc05178493","Type":"ContainerDied","Data":"1943ed034338fe95406594fcd052586adb2fe8cd83d72b98dddb58331aeccba1"}
Sep 11 18:42:19.153232 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:19.153202    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:42:20.226057 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:20.226033    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg"
Sep 11 18:42:20.248218 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:20.248195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50746007-2cbb-4739-8877-32dc05178493-maas-api-service-ca\") pod \"50746007-2cbb-4739-8877-32dc05178493\" (UID: \"50746007-2cbb-4739-8877-32dc05178493\") "
Sep 11 18:42:20.248579 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:20.248551    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50746007-2cbb-4739-8877-32dc05178493-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50746007-2cbb-4739-8877-32dc05178493" (UID: "50746007-2cbb-4739-8877-32dc05178493"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:42:20.349816 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:20.349743    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50746007-2cbb-4739-8877-32dc05178493-maas-api-service-ca\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:21.101205 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:21.101182    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg"
Sep 11 18:42:21.101205 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:21.101197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg" event={"ID":"50746007-2cbb-4739-8877-32dc05178493","Type":"ContainerDied","Data":"2de1d5e886f9ac419bf3f7c0cad6c1dfc66ea2650f38aed4f1f8ea2e0f48c4a9"}
Sep 11 18:42:21.101421 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:21.101237    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2de1d5e886f9ac419bf3f7c0cad6c1dfc66ea2650f38aed4f1f8ea2e0f48c4a9"
Sep 11 18:42:21.724554 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:21.724522    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"]
Sep 11 18:42:21.742815 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:21.742786    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"]
Sep 11 18:42:26.830850 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:26.830820    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"]
Sep 11 18:42:32.696187 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:32.696154    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"]
Sep 11 18:42:32.696663 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:32.696362    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj" podUID="f42b5999-d24c-46a2-9b2e-596e4680bcb8" containerName="payload-processing" containerID="cri-o://36daa507a65d65ddd62d3e2ec1723636e532f91c5881b1697861b3614f6db5f3" gracePeriod=30
Sep 11 18:42:32.704824 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:32.704800    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"]
Sep 11 18:42:32.705006 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:32.704972    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9" podUID="a8de682d-257c-4068-83a9-b3cb657ecddb" containerName="payload-pre-processing" containerID="cri-o://7db7e900d28aed8c895395748ba1e8bec29d718ac2c16d5f835fd8678fa00e9a" gracePeriod=30
Sep 11 18:42:32.895660 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:32.895629    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg"]
Sep 11 18:42:32.902530 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:32.902503    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-724327-vkcpg"]
Sep 11 18:42:32.976261 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:32.976242    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"
Sep 11 18:42:32.979324 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:32.979298    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"
Sep 11 18:42:33.066365 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.066333    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8de682d-257c-4068-83a9-b3cb657ecddb-plugins-config-volume\") pod \"a8de682d-257c-4068-83a9-b3cb657ecddb\" (UID: \"a8de682d-257c-4068-83a9-b3cb657ecddb\") "
Sep 11 18:42:33.066543 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.066376    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42b5999-d24c-46a2-9b2e-596e4680bcb8-plugins-config-volume\") pod \"f42b5999-d24c-46a2-9b2e-596e4680bcb8\" (UID: \"f42b5999-d24c-46a2-9b2e-596e4680bcb8\") "
Sep 11 18:42:33.066543 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.066505    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dpbxq\" (UniqueName: \"kubernetes.io/projected/a8de682d-257c-4068-83a9-b3cb657ecddb-kube-api-access-dpbxq\") pod \"a8de682d-257c-4068-83a9-b3cb657ecddb\" (UID: \"a8de682d-257c-4068-83a9-b3cb657ecddb\") "
Sep 11 18:42:33.066648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.066557    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5jp6\" (UniqueName: \"kubernetes.io/projected/f42b5999-d24c-46a2-9b2e-596e4680bcb8-kube-api-access-s5jp6\") pod \"f42b5999-d24c-46a2-9b2e-596e4680bcb8\" (UID: \"f42b5999-d24c-46a2-9b2e-596e4680bcb8\") "
Sep 11 18:42:33.066779 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.066752    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42b5999-d24c-46a2-9b2e-596e4680bcb8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f42b5999-d24c-46a2-9b2e-596e4680bcb8" (UID: "f42b5999-d24c-46a2-9b2e-596e4680bcb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:42:33.066779 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.066761    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8de682d-257c-4068-83a9-b3cb657ecddb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8de682d-257c-4068-83a9-b3cb657ecddb" (UID: "a8de682d-257c-4068-83a9-b3cb657ecddb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:42:33.066959 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.066870    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8de682d-257c-4068-83a9-b3cb657ecddb-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:33.066959 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.066890    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f42b5999-d24c-46a2-9b2e-596e4680bcb8-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:33.068549 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.068531    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8de682d-257c-4068-83a9-b3cb657ecddb-kube-api-access-dpbxq" (OuterVolumeSpecName: "kube-api-access-dpbxq") pod "a8de682d-257c-4068-83a9-b3cb657ecddb" (UID: "a8de682d-257c-4068-83a9-b3cb657ecddb"). InnerVolumeSpecName "kube-api-access-dpbxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:42:33.068639 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.068622    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f42b5999-d24c-46a2-9b2e-596e4680bcb8-kube-api-access-s5jp6" (OuterVolumeSpecName: "kube-api-access-s5jp6") pod "f42b5999-d24c-46a2-9b2e-596e4680bcb8" (UID: "f42b5999-d24c-46a2-9b2e-596e4680bcb8"). InnerVolumeSpecName "kube-api-access-s5jp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:42:33.166811 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.166775    2581 generic.go:356] "Generic (PLEG): container finished" podID="f42b5999-d24c-46a2-9b2e-596e4680bcb8" containerID="36daa507a65d65ddd62d3e2ec1723636e532f91c5881b1697861b3614f6db5f3" exitCode=0
Sep 11 18:42:33.166963 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.166825    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"
Sep 11 18:42:33.166963 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.166854    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj" event={"ID":"f42b5999-d24c-46a2-9b2e-596e4680bcb8","Type":"ContainerDied","Data":"36daa507a65d65ddd62d3e2ec1723636e532f91c5881b1697861b3614f6db5f3"}
Sep 11 18:42:33.166963 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.166904    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj" event={"ID":"f42b5999-d24c-46a2-9b2e-596e4680bcb8","Type":"ContainerDied","Data":"191721a19b848127bf0d2bb58eab2de3e3f5d2ed52b7043822845ef0831bc3ca"}
Sep 11 18:42:33.166963 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.166929    2581 scope.go:117] "RemoveContainer" containerID="36daa507a65d65ddd62d3e2ec1723636e532f91c5881b1697861b3614f6db5f3"
Sep 11 18:42:33.167427 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.167404    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpbxq\" (UniqueName: \"kubernetes.io/projected/a8de682d-257c-4068-83a9-b3cb657ecddb-kube-api-access-dpbxq\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:33.167589 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.167431    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5jp6\" (UniqueName: \"kubernetes.io/projected/f42b5999-d24c-46a2-9b2e-596e4680bcb8-kube-api-access-s5jp6\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:33.168411 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.168388    2581 generic.go:356] "Generic (PLEG): container finished" podID="a8de682d-257c-4068-83a9-b3cb657ecddb" containerID="7db7e900d28aed8c895395748ba1e8bec29d718ac2c16d5f835fd8678fa00e9a" exitCode=0
Sep 11 18:42:33.168548 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.168424    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"
Sep 11 18:42:33.168548 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.168435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9" event={"ID":"a8de682d-257c-4068-83a9-b3cb657ecddb","Type":"ContainerDied","Data":"7db7e900d28aed8c895395748ba1e8bec29d718ac2c16d5f835fd8678fa00e9a"}
Sep 11 18:42:33.168548 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.168465    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9" event={"ID":"a8de682d-257c-4068-83a9-b3cb657ecddb","Type":"ContainerDied","Data":"7ebc82cea3402952e2cbbae41c64805b8e321b6f4f8eb76834f7fd315fc8059b"}
Sep 11 18:42:33.175859 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.175845    2581 scope.go:117] "RemoveContainer" containerID="36daa507a65d65ddd62d3e2ec1723636e532f91c5881b1697861b3614f6db5f3"
Sep 11 18:42:33.176129 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:42:33.176112    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36daa507a65d65ddd62d3e2ec1723636e532f91c5881b1697861b3614f6db5f3\": container with ID starting with 36daa507a65d65ddd62d3e2ec1723636e532f91c5881b1697861b3614f6db5f3 not found: ID does not exist" containerID="36daa507a65d65ddd62d3e2ec1723636e532f91c5881b1697861b3614f6db5f3"
Sep 11 18:42:33.176182 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.176137    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36daa507a65d65ddd62d3e2ec1723636e532f91c5881b1697861b3614f6db5f3"} err="failed to get container status \"36daa507a65d65ddd62d3e2ec1723636e532f91c5881b1697861b3614f6db5f3\": rpc error: code = NotFound desc = could not find container \"36daa507a65d65ddd62d3e2ec1723636e532f91c5881b1697861b3614f6db5f3\": container with ID starting with 36daa507a65d65ddd62d3e2ec1723636e532f91c5881b1697861b3614f6db5f3 not found: ID does not exist"
Sep 11 18:42:33.176182 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.176152    2581 scope.go:117] "RemoveContainer" containerID="7db7e900d28aed8c895395748ba1e8bec29d718ac2c16d5f835fd8678fa00e9a"
Sep 11 18:42:33.184054 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.184040    2581 scope.go:117] "RemoveContainer" containerID="7db7e900d28aed8c895395748ba1e8bec29d718ac2c16d5f835fd8678fa00e9a"
Sep 11 18:42:33.184278 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:42:33.184263    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7db7e900d28aed8c895395748ba1e8bec29d718ac2c16d5f835fd8678fa00e9a\": container with ID starting with 7db7e900d28aed8c895395748ba1e8bec29d718ac2c16d5f835fd8678fa00e9a not found: ID does not exist" containerID="7db7e900d28aed8c895395748ba1e8bec29d718ac2c16d5f835fd8678fa00e9a"
Sep 11 18:42:33.184323 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.184283    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7db7e900d28aed8c895395748ba1e8bec29d718ac2c16d5f835fd8678fa00e9a"} err="failed to get container status \"7db7e900d28aed8c895395748ba1e8bec29d718ac2c16d5f835fd8678fa00e9a\": rpc error: code = NotFound desc = could not find container \"7db7e900d28aed8c895395748ba1e8bec29d718ac2c16d5f835fd8678fa00e9a\": container with ID starting with 7db7e900d28aed8c895395748ba1e8bec29d718ac2c16d5f835fd8678fa00e9a not found: ID does not exist"
Sep 11 18:42:33.191661 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.191641    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"]
Sep 11 18:42:33.204083 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.204040    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-724327-7548d67f49-69mpj"]
Sep 11 18:42:33.216012 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.215992    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"]
Sep 11 18:42:33.221582 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.221563    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-724327-774f796c8-fnbj9"]
Sep 11 18:42:33.764645 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.764610    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50746007-2cbb-4739-8877-32dc05178493" path="/var/lib/kubelet/pods/50746007-2cbb-4739-8877-32dc05178493/volumes"
Sep 11 18:42:33.764979 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.764960    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8de682d-257c-4068-83a9-b3cb657ecddb" path="/var/lib/kubelet/pods/a8de682d-257c-4068-83a9-b3cb657ecddb/volumes"
Sep 11 18:42:33.765278 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:33.765264    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f42b5999-d24c-46a2-9b2e-596e4680bcb8" path="/var/lib/kubelet/pods/f42b5999-d24c-46a2-9b2e-596e4680bcb8/volumes"
Sep 11 18:42:36.744684 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:36.744623    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz" podUID="2bb583ad-8821-4669-b229-7e57b8b94add" containerName="main" containerID="cri-o://51011e755f52ba121d18f7f83ffcfc2bbda47eb3a50f512a5e5b0168d2c26d5f" gracePeriod=45
Sep 11 18:42:36.763835 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:36.763806    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k" podUID="87de27b2-081b-4e99-b864-dcd612cc4261" containerName="main" containerID="cri-o://c93a9bc856cd2261b64a54fbbeb10c2b7c4420c30aca037177ffc855213bb1a5" gracePeriod=45
Sep 11 18:42:37.010499 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.010424    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:42:37.040055 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.040032    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:42:37.098131 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.098092    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-model-cache\") pod \"2bb583ad-8821-4669-b229-7e57b8b94add\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") "
Sep 11 18:42:37.098330 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.098151    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-dshm\") pod \"87de27b2-081b-4e99-b864-dcd612cc4261\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") "
Sep 11 18:42:37.098330 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.098193    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-tmp-dir\") pod \"87de27b2-081b-4e99-b864-dcd612cc4261\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") "
Sep 11 18:42:37.098330 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.098238    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87de27b2-081b-4e99-b864-dcd612cc4261-tls-certs\") pod \"87de27b2-081b-4e99-b864-dcd612cc4261\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") "
Sep 11 18:42:37.098330 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.098315    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkgh6\" (UniqueName: \"kubernetes.io/projected/87de27b2-081b-4e99-b864-dcd612cc4261-kube-api-access-tkgh6\") pod \"87de27b2-081b-4e99-b864-dcd612cc4261\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") "
Sep 11 18:42:37.098594 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.098339    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btgrj\" (UniqueName: \"kubernetes.io/projected/2bb583ad-8821-4669-b229-7e57b8b94add-kube-api-access-btgrj\") pod \"2bb583ad-8821-4669-b229-7e57b8b94add\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") "
Sep 11 18:42:37.098594 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.098365    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-tmp-dir\") pod \"2bb583ad-8821-4669-b229-7e57b8b94add\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") "
Sep 11 18:42:37.098594 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.098448    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87de27b2-081b-4e99-b864-dcd612cc4261" (UID: "87de27b2-081b-4e99-b864-dcd612cc4261"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:37.098757 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.098604    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2bb583ad-8821-4669-b229-7e57b8b94add" (UID: "2bb583ad-8821-4669-b229-7e57b8b94add"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:37.098757 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.098626    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-model-cache" (OuterVolumeSpecName: "model-cache") pod "2bb583ad-8821-4669-b229-7e57b8b94add" (UID: "2bb583ad-8821-4669-b229-7e57b8b94add"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:37.098757 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.098718    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-model-cache\") pod \"87de27b2-081b-4e99-b864-dcd612cc4261\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") "
Sep 11 18:42:37.098918 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.098786    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-home\") pod \"87de27b2-081b-4e99-b864-dcd612cc4261\" (UID: \"87de27b2-081b-4e99-b864-dcd612cc4261\") "
Sep 11 18:42:37.098918 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.098838    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-dshm\") pod \"2bb583ad-8821-4669-b229-7e57b8b94add\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") "
Sep 11 18:42:37.098918 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.098875    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-home\") pod \"2bb583ad-8821-4669-b229-7e57b8b94add\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") "
Sep 11 18:42:37.098918 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.098915    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bb583ad-8821-4669-b229-7e57b8b94add-tls-certs\") pod \"2bb583ad-8821-4669-b229-7e57b8b94add\" (UID: \"2bb583ad-8821-4669-b229-7e57b8b94add\") "
Sep 11 18:42:37.099176 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.099036    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-home" (OuterVolumeSpecName: "home") pod "87de27b2-081b-4e99-b864-dcd612cc4261" (UID: "87de27b2-081b-4e99-b864-dcd612cc4261"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:37.099176 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.099177    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-home" (OuterVolumeSpecName: "home") pod "2bb583ad-8821-4669-b229-7e57b8b94add" (UID: "2bb583ad-8821-4669-b229-7e57b8b94add"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:37.099404 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.099305    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.099404 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.099324    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.099404 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.099337    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.099404 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.099349    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.099404 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.099361    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.099693 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.099583    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-model-cache" (OuterVolumeSpecName: "model-cache") pod "87de27b2-081b-4e99-b864-dcd612cc4261" (UID: "87de27b2-081b-4e99-b864-dcd612cc4261"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:37.101261 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.101188    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb583ad-8821-4669-b229-7e57b8b94add-kube-api-access-btgrj" (OuterVolumeSpecName: "kube-api-access-btgrj") pod "2bb583ad-8821-4669-b229-7e57b8b94add" (UID: "2bb583ad-8821-4669-b229-7e57b8b94add"). InnerVolumeSpecName "kube-api-access-btgrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:42:37.101426 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.101301    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-dshm" (OuterVolumeSpecName: "dshm") pod "87de27b2-081b-4e99-b864-dcd612cc4261" (UID: "87de27b2-081b-4e99-b864-dcd612cc4261"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:37.101634 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.101617    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-dshm" (OuterVolumeSpecName: "dshm") pod "2bb583ad-8821-4669-b229-7e57b8b94add" (UID: "2bb583ad-8821-4669-b229-7e57b8b94add"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:37.101758 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.101731    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb583ad-8821-4669-b229-7e57b8b94add-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2bb583ad-8821-4669-b229-7e57b8b94add" (UID: "2bb583ad-8821-4669-b229-7e57b8b94add"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:42:37.101816 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.101800    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87de27b2-081b-4e99-b864-dcd612cc4261-kube-api-access-tkgh6" (OuterVolumeSpecName: "kube-api-access-tkgh6") pod "87de27b2-081b-4e99-b864-dcd612cc4261" (UID: "87de27b2-081b-4e99-b864-dcd612cc4261"). InnerVolumeSpecName "kube-api-access-tkgh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:42:37.103140 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.103119    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87de27b2-081b-4e99-b864-dcd612cc4261-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87de27b2-081b-4e99-b864-dcd612cc4261" (UID: "87de27b2-081b-4e99-b864-dcd612cc4261"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:42:37.192307 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.192272    2581 generic.go:356] "Generic (PLEG): container finished" podID="2bb583ad-8821-4669-b229-7e57b8b94add" containerID="51011e755f52ba121d18f7f83ffcfc2bbda47eb3a50f512a5e5b0168d2c26d5f" exitCode=0
Sep 11 18:42:37.192470 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.192317    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"
Sep 11 18:42:37.192470 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.192356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz" event={"ID":"2bb583ad-8821-4669-b229-7e57b8b94add","Type":"ContainerDied","Data":"51011e755f52ba121d18f7f83ffcfc2bbda47eb3a50f512a5e5b0168d2c26d5f"}
Sep 11 18:42:37.192470 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.192401    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz" event={"ID":"2bb583ad-8821-4669-b229-7e57b8b94add","Type":"ContainerDied","Data":"83144454850ab6fbfe0322f841b3f453a74ff15829b13d3d8abd7520d54b9e9e"}
Sep 11 18:42:37.192470 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.192417    2581 scope.go:117] "RemoveContainer" containerID="51011e755f52ba121d18f7f83ffcfc2bbda47eb3a50f512a5e5b0168d2c26d5f"
Sep 11 18:42:37.193918 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.193892    2581 generic.go:356] "Generic (PLEG): container finished" podID="87de27b2-081b-4e99-b864-dcd612cc4261" containerID="c93a9bc856cd2261b64a54fbbeb10c2b7c4420c30aca037177ffc855213bb1a5" exitCode=0
Sep 11 18:42:37.194035 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.193928    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"
Sep 11 18:42:37.194035 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.193932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k" event={"ID":"87de27b2-081b-4e99-b864-dcd612cc4261","Type":"ContainerDied","Data":"c93a9bc856cd2261b64a54fbbeb10c2b7c4420c30aca037177ffc855213bb1a5"}
Sep 11 18:42:37.194035 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.193977    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k" event={"ID":"87de27b2-081b-4e99-b864-dcd612cc4261","Type":"ContainerDied","Data":"3b627defe241b169fb621d8af8be1c39545df8e134858d1ed0ab8b72b5e298d7"}
Sep 11 18:42:37.199859 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.199838    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.199942 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.199863    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87de27b2-081b-4e99-b864-dcd612cc4261-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.199942 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.199874    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkgh6\" (UniqueName: \"kubernetes.io/projected/87de27b2-081b-4e99-b864-dcd612cc4261-kube-api-access-tkgh6\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.199942 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.199883    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btgrj\" (UniqueName: \"kubernetes.io/projected/2bb583ad-8821-4669-b229-7e57b8b94add-kube-api-access-btgrj\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.199942 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.199891    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87de27b2-081b-4e99-b864-dcd612cc4261-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.199942 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.199898    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bb583ad-8821-4669-b229-7e57b8b94add-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.199942 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.199905    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bb583ad-8821-4669-b229-7e57b8b94add-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:37.202924 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.202895    2581 scope.go:117] "RemoveContainer" containerID="51011e755f52ba121d18f7f83ffcfc2bbda47eb3a50f512a5e5b0168d2c26d5f"
Sep 11 18:42:37.203186 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:42:37.203162    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51011e755f52ba121d18f7f83ffcfc2bbda47eb3a50f512a5e5b0168d2c26d5f\": container with ID starting with 51011e755f52ba121d18f7f83ffcfc2bbda47eb3a50f512a5e5b0168d2c26d5f not found: ID does not exist" containerID="51011e755f52ba121d18f7f83ffcfc2bbda47eb3a50f512a5e5b0168d2c26d5f"
Sep 11 18:42:37.203269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.203192    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51011e755f52ba121d18f7f83ffcfc2bbda47eb3a50f512a5e5b0168d2c26d5f"} err="failed to get container status \"51011e755f52ba121d18f7f83ffcfc2bbda47eb3a50f512a5e5b0168d2c26d5f\": rpc error: code = NotFound desc = could not find container \"51011e755f52ba121d18f7f83ffcfc2bbda47eb3a50f512a5e5b0168d2c26d5f\": container with ID starting with 51011e755f52ba121d18f7f83ffcfc2bbda47eb3a50f512a5e5b0168d2c26d5f not found: ID does not exist"
Sep 11 18:42:37.203269 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.203208    2581 scope.go:117] "RemoveContainer" containerID="c93a9bc856cd2261b64a54fbbeb10c2b7c4420c30aca037177ffc855213bb1a5"
Sep 11 18:42:37.215078 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.214283    2581 scope.go:117] "RemoveContainer" containerID="c93a9bc856cd2261b64a54fbbeb10c2b7c4420c30aca037177ffc855213bb1a5"
Sep 11 18:42:37.215078 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:42:37.214844    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93a9bc856cd2261b64a54fbbeb10c2b7c4420c30aca037177ffc855213bb1a5\": container with ID starting with c93a9bc856cd2261b64a54fbbeb10c2b7c4420c30aca037177ffc855213bb1a5 not found: ID does not exist" containerID="c93a9bc856cd2261b64a54fbbeb10c2b7c4420c30aca037177ffc855213bb1a5"
Sep 11 18:42:37.215078 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.214876    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93a9bc856cd2261b64a54fbbeb10c2b7c4420c30aca037177ffc855213bb1a5"} err="failed to get container status \"c93a9bc856cd2261b64a54fbbeb10c2b7c4420c30aca037177ffc855213bb1a5\": rpc error: code = NotFound desc = could not find container \"c93a9bc856cd2261b64a54fbbeb10c2b7c4420c30aca037177ffc855213bb1a5\": container with ID starting with c93a9bc856cd2261b64a54fbbeb10c2b7c4420c30aca037177ffc855213bb1a5 not found: ID does not exist"
Sep 11 18:42:37.216733 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.216714    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"]
Sep 11 18:42:37.227035 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.227008    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-724327/distinct-w4-724327-kserve-85d76d749b-5nxqz"]
Sep 11 18:42:37.238098 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.238075    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"]
Sep 11 18:42:37.250257 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.250235    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-724327/premium-w4-724327-kserve-d68bc4957-xkp5k"]
Sep 11 18:42:37.763675 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.763650    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bb583ad-8821-4669-b229-7e57b8b94add" path="/var/lib/kubelet/pods/2bb583ad-8821-4669-b229-7e57b8b94add/volumes"
Sep 11 18:42:37.764029 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:37.764016    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87de27b2-081b-4e99-b864-dcd612cc4261" path="/var/lib/kubelet/pods/87de27b2-081b-4e99-b864-dcd612cc4261/volumes"
Sep 11 18:42:39.243290 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:39.243258    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:42:41.850242 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:41.850197    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr" podUID="aa5bf91d-ab50-4628-98ad-0974d5d92b5a" containerName="main" containerID="cri-o://cec6ffd8f036a750c3380b791b9a0dabffde761b9484467ef716520bcdfb2da7" gracePeriod=45
Sep 11 18:42:42.101654 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.101597    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:42:42.223816 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.223780    2581 generic.go:356] "Generic (PLEG): container finished" podID="aa5bf91d-ab50-4628-98ad-0974d5d92b5a" containerID="cec6ffd8f036a750c3380b791b9a0dabffde761b9484467ef716520bcdfb2da7" exitCode=0
Sep 11 18:42:42.223816 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.223819    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"
Sep 11 18:42:42.224069 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.223866    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr" event={"ID":"aa5bf91d-ab50-4628-98ad-0974d5d92b5a","Type":"ContainerDied","Data":"cec6ffd8f036a750c3380b791b9a0dabffde761b9484467ef716520bcdfb2da7"}
Sep 11 18:42:42.224069 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.223910    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr" event={"ID":"aa5bf91d-ab50-4628-98ad-0974d5d92b5a","Type":"ContainerDied","Data":"7cb7d898f8483b9a031313d03baaaf7b65ebe02c8614b7ce52273a6df55bb080"}
Sep 11 18:42:42.224069 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.223932    2581 scope.go:117] "RemoveContainer" containerID="cec6ffd8f036a750c3380b791b9a0dabffde761b9484467ef716520bcdfb2da7"
Sep 11 18:42:42.232446 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.232429    2581 scope.go:117] "RemoveContainer" containerID="cec6ffd8f036a750c3380b791b9a0dabffde761b9484467ef716520bcdfb2da7"
Sep 11 18:42:42.232711 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:42:42.232693    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec6ffd8f036a750c3380b791b9a0dabffde761b9484467ef716520bcdfb2da7\": container with ID starting with cec6ffd8f036a750c3380b791b9a0dabffde761b9484467ef716520bcdfb2da7 not found: ID does not exist" containerID="cec6ffd8f036a750c3380b791b9a0dabffde761b9484467ef716520bcdfb2da7"
Sep 11 18:42:42.232775 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.232718    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec6ffd8f036a750c3380b791b9a0dabffde761b9484467ef716520bcdfb2da7"} err="failed to get container status \"cec6ffd8f036a750c3380b791b9a0dabffde761b9484467ef716520bcdfb2da7\": rpc error: code = NotFound desc = could not find container \"cec6ffd8f036a750c3380b791b9a0dabffde761b9484467ef716520bcdfb2da7\": container with ID starting with cec6ffd8f036a750c3380b791b9a0dabffde761b9484467ef716520bcdfb2da7 not found: ID does not exist"
Sep 11 18:42:42.245037 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.245019    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-dshm\") pod \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") "
Sep 11 18:42:42.245130 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.245049    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-tls-certs\") pod \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") "
Sep 11 18:42:42.245130 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.245067    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k8b8\" (UniqueName: \"kubernetes.io/projected/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-kube-api-access-8k8b8\") pod \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") "
Sep 11 18:42:42.245130 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.245083    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-tmp-dir\") pod \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") "
Sep 11 18:42:42.245130 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.245121    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-model-cache\") pod \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") "
Sep 11 18:42:42.245335 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.245195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-home\") pod \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\" (UID: \"aa5bf91d-ab50-4628-98ad-0974d5d92b5a\") "
Sep 11 18:42:42.245393 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.245328    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa5bf91d-ab50-4628-98ad-0974d5d92b5a" (UID: "aa5bf91d-ab50-4628-98ad-0974d5d92b5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:42.245450 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.245398    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa5bf91d-ab50-4628-98ad-0974d5d92b5a" (UID: "aa5bf91d-ab50-4628-98ad-0974d5d92b5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:42.245537 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.245466    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-home" (OuterVolumeSpecName: "home") pod "aa5bf91d-ab50-4628-98ad-0974d5d92b5a" (UID: "aa5bf91d-ab50-4628-98ad-0974d5d92b5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:42.245584 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.245575    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:42.245619 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.245590    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:42.245619 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.245604    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:42.247274 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.247257    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa5bf91d-ab50-4628-98ad-0974d5d92b5a" (UID: "aa5bf91d-ab50-4628-98ad-0974d5d92b5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:42:42.247667 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.247648    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-dshm" (OuterVolumeSpecName: "dshm") pod "aa5bf91d-ab50-4628-98ad-0974d5d92b5a" (UID: "aa5bf91d-ab50-4628-98ad-0974d5d92b5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:42:42.247728 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.247690    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-kube-api-access-8k8b8" (OuterVolumeSpecName: "kube-api-access-8k8b8") pod "aa5bf91d-ab50-4628-98ad-0974d5d92b5a" (UID: "aa5bf91d-ab50-4628-98ad-0974d5d92b5a"). InnerVolumeSpecName "kube-api-access-8k8b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:42:42.346572 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.346544    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:42.346572 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.346567    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:42.346572 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.346577    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k8b8\" (UniqueName: \"kubernetes.io/projected/aa5bf91d-ab50-4628-98ad-0974d5d92b5a-kube-api-access-8k8b8\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:42:42.546524 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.546501    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"]
Sep 11 18:42:42.555034 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:42.555010    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/auth-test-model-711a34-kserve-7f4bfbb66c-l5dcr"]
Sep 11 18:42:43.764107 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:43.764077    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa5bf91d-ab50-4628-98ad-0974d5d92b5a" path="/var/lib/kubelet/pods/aa5bf91d-ab50-4628-98ad-0974d5d92b5a/volumes"
Sep 11 18:42:57.357985 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.357953    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"]
Sep 11 18:42:57.358373 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358344    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87de27b2-081b-4e99-b864-dcd612cc4261" containerName="main"
Sep 11 18:42:57.358373 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358356    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="87de27b2-081b-4e99-b864-dcd612cc4261" containerName="main"
Sep 11 18:42:57.358373 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358370    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa5bf91d-ab50-4628-98ad-0974d5d92b5a" containerName="main"
Sep 11 18:42:57.358373 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358375    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa5bf91d-ab50-4628-98ad-0974d5d92b5a" containerName="main"
Sep 11 18:42:57.358521 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358383    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bb583ad-8821-4669-b229-7e57b8b94add" containerName="main"
Sep 11 18:42:57.358521 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358388    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bb583ad-8821-4669-b229-7e57b8b94add" containerName="main"
Sep 11 18:42:57.358521 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358399    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8de682d-257c-4068-83a9-b3cb657ecddb" containerName="payload-pre-processing"
Sep 11 18:42:57.358521 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358405    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8de682d-257c-4068-83a9-b3cb657ecddb" containerName="payload-pre-processing"
Sep 11 18:42:57.358521 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358416    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f42b5999-d24c-46a2-9b2e-596e4680bcb8" containerName="payload-processing"
Sep 11 18:42:57.358521 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358421    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f42b5999-d24c-46a2-9b2e-596e4680bcb8" containerName="payload-processing"
Sep 11 18:42:57.358521 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358427    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50746007-2cbb-4739-8877-32dc05178493" containerName="revoke-keys"
Sep 11 18:42:57.358521 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358432    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="50746007-2cbb-4739-8877-32dc05178493" containerName="revoke-keys"
Sep 11 18:42:57.358758 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358530    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f42b5999-d24c-46a2-9b2e-596e4680bcb8" containerName="payload-processing"
Sep 11 18:42:57.358758 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358541    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="87de27b2-081b-4e99-b864-dcd612cc4261" containerName="main"
Sep 11 18:42:57.358758 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358547    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bb583ad-8821-4669-b229-7e57b8b94add" containerName="main"
Sep 11 18:42:57.358758 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358553    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa5bf91d-ab50-4628-98ad-0974d5d92b5a" containerName="main"
Sep 11 18:42:57.358758 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358558    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8de682d-257c-4068-83a9-b3cb657ecddb" containerName="payload-pre-processing"
Sep 11 18:42:57.358758 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.358563    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="50746007-2cbb-4739-8877-32dc05178493" containerName="revoke-keys"
Sep 11 18:42:57.361603 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.361588    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.364410 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.364387    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7ca6df\"/\"sub-test-model-7ca6df-kserve-self-signed-certs\""
Sep 11 18:42:57.364410 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.364387    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7ca6df\"/\"openshift-service-ca.crt\""
Sep 11 18:42:57.364646 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.364432    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-7ca6df\"/\"kube-root-ca.crt\""
Sep 11 18:42:57.364646 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.364387    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-7ca6df\"/\"default-dockercfg-9lxk4\""
Sep 11 18:42:57.372691 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.372670    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"]
Sep 11 18:42:57.493055 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.493030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-dshm\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.493202 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.493103    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz797\" (UniqueName: \"kubernetes.io/projected/7803e616-73e7-4e43-a404-90e336546bbd-kube-api-access-lz797\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.493202 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.493127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-home\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.493202 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.493147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-model-cache\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.493202 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.493172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7803e616-73e7-4e43-a404-90e336546bbd-tls-certs\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.493202 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.493189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-tmp-dir\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.594407 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.594381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz797\" (UniqueName: \"kubernetes.io/projected/7803e616-73e7-4e43-a404-90e336546bbd-kube-api-access-lz797\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.594613 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.594411    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-home\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.594613 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.594433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-model-cache\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.594613 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.594458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7803e616-73e7-4e43-a404-90e336546bbd-tls-certs\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.594613 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.594495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-tmp-dir\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.594613 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.594546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-dshm\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.594887 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.594792    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-home\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.594887 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.594876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-model-cache\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.594967 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.594882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-tmp-dir\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.596886 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.596867    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-dshm\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.597120 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.597105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7803e616-73e7-4e43-a404-90e336546bbd-tls-certs\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.606036 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.606017    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz797\" (UniqueName: \"kubernetes.io/projected/7803e616-73e7-4e43-a404-90e336546bbd-kube-api-access-lz797\") pod \"sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") " pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.670337 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.670319    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:42:57.939873 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:57.939850    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"]
Sep 11 18:42:57.941954 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:42:57.941928    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7803e616_73e7_4e43_a404_90e336546bbd.slice/crio-3660ffbafb8a25c52328f0c4025a3b0f31eced5bba699cf9cb1099850f0ee104 WatchSource:0}: Error finding container 3660ffbafb8a25c52328f0c4025a3b0f31eced5bba699cf9cb1099850f0ee104: Status 404 returned error can't find the container with id 3660ffbafb8a25c52328f0c4025a3b0f31eced5bba699cf9cb1099850f0ee104
Sep 11 18:42:58.302727 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:58.302635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb" event={"ID":"7803e616-73e7-4e43-a404-90e336546bbd","Type":"ContainerStarted","Data":"355d68b2bb46e8feed1c339943631530809579814a37a32613b660aeed682a01"}
Sep 11 18:42:58.302727 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:58.302689    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb" event={"ID":"7803e616-73e7-4e43-a404-90e336546bbd","Type":"ContainerStarted","Data":"3660ffbafb8a25c52328f0c4025a3b0f31eced5bba699cf9cb1099850f0ee104"}
Sep 11 18:42:58.322872 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:42:58.322828    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb" podStartSLOduration=1.322813241 podStartE2EDuration="1.322813241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:42:58.321367518 +0000 UTC m=+1655.062494373" watchObservedRunningTime="2026-09-11 18:42:58.322813241 +0000 UTC m=+1655.063940082"
Sep 11 18:43:00.958087 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:00.958052    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:43:07.671140 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:07.671103    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:43:07.671512 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:07.671163    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:43:07.684079 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:07.684049    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:43:08.364973 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:08.364945    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:43:23.982756 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:23.982727    2581 scope.go:117] "RemoveContainer" containerID="004cb877ba59c46363f65d49bdba1d5daf2abfd361f6fc7121830cd2fcaecb87"
Sep 11 18:43:35.051577 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.051543    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"]
Sep 11 18:43:35.056543 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.056521    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.059540 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.059515    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-711a34\"/\"kube-root-ca.crt\""
Sep 11 18:43:35.059673 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.059558    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-711a34\"/\"default-dockercfg-f7z6v\""
Sep 11 18:43:35.059673 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.059607    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-711a34\"/\"openshift-service-ca.crt\""
Sep 11 18:43:35.059673 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.059657    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-711a34\"/\"sub-test-model-711a34-kserve-self-signed-certs\""
Sep 11 18:43:35.065609 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.065587    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"]
Sep 11 18:43:35.115072 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.115033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-home\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.115072 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.115067    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-dshm\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.115297 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.115090    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4920645c-66f7-4296-b236-c708ae4827df-tls-certs\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.115297 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.115199    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxt8r\" (UniqueName: \"kubernetes.io/projected/4920645c-66f7-4296-b236-c708ae4827df-kube-api-access-kxt8r\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.115297 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.115240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-model-cache\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.115297 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.115261    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-tmp-dir\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.216705 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.216674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxt8r\" (UniqueName: \"kubernetes.io/projected/4920645c-66f7-4296-b236-c708ae4827df-kube-api-access-kxt8r\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.216873 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.216714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-model-cache\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.216873 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.216740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-tmp-dir\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.216873 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.216809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-home\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.216873 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.216832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-dshm\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.216873 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.216848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4920645c-66f7-4296-b236-c708ae4827df-tls-certs\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.217151 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.217113    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-model-cache\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.217151 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.217137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-tmp-dir\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.217230 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.217166    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-home\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.219139 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.219119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-dshm\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.219379 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.219364    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4920645c-66f7-4296-b236-c708ae4827df-tls-certs\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.227910 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.227892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxt8r\" (UniqueName: \"kubernetes.io/projected/4920645c-66f7-4296-b236-c708ae4827df-kube-api-access-kxt8r\") pod \"sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") " pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.367110 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.367037    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:35.639592 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:35.639567    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"]
Sep 11 18:43:35.641529 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:43:35.641504    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4920645c_66f7_4296_b236_c708ae4827df.slice/crio-14f971ab00183b63420173223f4ab6d3b2188c9702d8dc6ef65a599c9f888e8e WatchSource:0}: Error finding container 14f971ab00183b63420173223f4ab6d3b2188c9702d8dc6ef65a599c9f888e8e: Status 404 returned error can't find the container with id 14f971ab00183b63420173223f4ab6d3b2188c9702d8dc6ef65a599c9f888e8e
Sep 11 18:43:36.500703 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:36.500668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv" event={"ID":"4920645c-66f7-4296-b236-c708ae4827df","Type":"ContainerStarted","Data":"98b5f1788a2740d89769677c815d232c050b33bd2fb40eacc183cb4b925d41fb"}
Sep 11 18:43:36.500703 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:36.500708    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv" event={"ID":"4920645c-66f7-4296-b236-c708ae4827df","Type":"ContainerStarted","Data":"14f971ab00183b63420173223f4ab6d3b2188c9702d8dc6ef65a599c9f888e8e"}
Sep 11 18:43:36.517805 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:36.517759    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv" podStartSLOduration=1.517742844 podStartE2EDuration="1.517742844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:43:36.515864336 +0000 UTC m=+1693.256991180" watchObservedRunningTime="2026-09-11 18:43:36.517742844 +0000 UTC m=+1693.258869687"
Sep 11 18:43:38.748620 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:38.748583    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:43:45.368052 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:45.367975    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:45.368052 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:45.368022    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:45.380887 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:45.380861    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:43:45.558467 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:43:45.558433    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:44:04.549517 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:04.549468    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:44:14.340669 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:14.340634    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:44:18.253176 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:18.253146    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7638-78e927"]
Sep 11 18:44:18.257074 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:18.257057    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-78e927"
Sep 11 18:44:18.259067 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:18.259044    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bbqnn\""
Sep 11 18:44:18.263037 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:18.263017    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7638-78e927"]
Sep 11 18:44:18.324066 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:18.324038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh257\" (UniqueName: \"kubernetes.io/projected/4d92766b-be44-4cc0-ae33-6d8574f4f604-kube-api-access-wh257\") pod \"test-curl-7638-78e927\" (UID: \"4d92766b-be44-4cc0-ae33-6d8574f4f604\") " pod="opendatahub/test-curl-7638-78e927"
Sep 11 18:44:18.425442 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:18.425410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh257\" (UniqueName: \"kubernetes.io/projected/4d92766b-be44-4cc0-ae33-6d8574f4f604-kube-api-access-wh257\") pod \"test-curl-7638-78e927\" (UID: \"4d92766b-be44-4cc0-ae33-6d8574f4f604\") " pod="opendatahub/test-curl-7638-78e927"
Sep 11 18:44:18.437203 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:18.437183    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh257\" (UniqueName: \"kubernetes.io/projected/4d92766b-be44-4cc0-ae33-6d8574f4f604-kube-api-access-wh257\") pod \"test-curl-7638-78e927\" (UID: \"4d92766b-be44-4cc0-ae33-6d8574f4f604\") " pod="opendatahub/test-curl-7638-78e927"
Sep 11 18:44:18.569371 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:18.569306    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-78e927"
Sep 11 18:44:18.836793 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:18.836767    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7638-78e927"]
Sep 11 18:44:18.839304 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:44:18.839277    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d92766b_be44_4cc0_ae33_6d8574f4f604.slice/crio-86ff1c587d8de52724f0311aa10cf5ca9809a2606486371b55aa30a1469b65e5 WatchSource:0}: Error finding container 86ff1c587d8de52724f0311aa10cf5ca9809a2606486371b55aa30a1469b65e5: Status 404 returned error can't find the container with id 86ff1c587d8de52724f0311aa10cf5ca9809a2606486371b55aa30a1469b65e5
Sep 11 18:44:19.720547 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:19.720511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7638-78e927" event={"ID":"4d92766b-be44-4cc0-ae33-6d8574f4f604","Type":"ContainerStarted","Data":"ad5ed06edf4e499486aea915bb0a611dc3d97bf514a8c61c19a621f7f7ebd6df"}
Sep 11 18:44:19.721000 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:19.720557    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7638-78e927" event={"ID":"4d92766b-be44-4cc0-ae33-6d8574f4f604","Type":"ContainerStarted","Data":"86ff1c587d8de52724f0311aa10cf5ca9809a2606486371b55aa30a1469b65e5"}
Sep 11 18:44:19.737260 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:19.737211    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7638-78e927" podStartSLOduration=1.737192361 podStartE2EDuration="1.737192361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:44:19.736581193 +0000 UTC m=+1736.477708036" watchObservedRunningTime="2026-09-11 18:44:19.737192361 +0000 UTC m=+1736.478319215"
Sep 11 18:44:20.338121 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:20.338086    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7638-78e927"]
Sep 11 18:44:20.340571 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:20.340544    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7638-78e927"]
Sep 11 18:44:20.726642 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:44:20.726614    2581 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7638-78e927" container="test-curl-7638-78e927"
Sep 11 18:44:21.731084 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:21.731030    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7638-78e927" podUID="4d92766b-be44-4cc0-ae33-6d8574f4f604" containerName="test-curl-7638-78e927" containerID="cri-o://ad5ed06edf4e499486aea915bb0a611dc3d97bf514a8c61c19a621f7f7ebd6df" gracePeriod=2
Sep 11 18:44:21.733044 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:44:21.733016    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7638-78e927\" is forbidden: User \"system:node:ip-10-0-129-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-166.ec2.internal' and this object" podUID="4d92766b-be44-4cc0-ae33-6d8574f4f604" pod="opendatahub/test-curl-7638-78e927"
Sep 11 18:44:22.334364 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:22.334331    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"]
Sep 11 18:44:23.286135 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:23.286100    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"]
Sep 11 18:44:23.913955 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:23.913936    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-78e927"
Sep 11 18:44:23.915862 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:44:23.915839    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7638-78e927\" is forbidden: User \"system:node:ip-10-0-129-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-166.ec2.internal' and this object" podUID="4d92766b-be44-4cc0-ae33-6d8574f4f604" pod="opendatahub/test-curl-7638-78e927"
Sep 11 18:44:24.073262 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:24.073193    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh257\" (UniqueName: \"kubernetes.io/projected/4d92766b-be44-4cc0-ae33-6d8574f4f604-kube-api-access-wh257\") pod \"4d92766b-be44-4cc0-ae33-6d8574f4f604\" (UID: \"4d92766b-be44-4cc0-ae33-6d8574f4f604\") "
Sep 11 18:44:24.075321 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:24.075294    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d92766b-be44-4cc0-ae33-6d8574f4f604-kube-api-access-wh257" (OuterVolumeSpecName: "kube-api-access-wh257") pod "4d92766b-be44-4cc0-ae33-6d8574f4f604" (UID: "4d92766b-be44-4cc0-ae33-6d8574f4f604"). InnerVolumeSpecName "kube-api-access-wh257". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:24.174580 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:24.174554    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh257\" (UniqueName: \"kubernetes.io/projected/4d92766b-be44-4cc0-ae33-6d8574f4f604-kube-api-access-wh257\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:44:24.749812 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:24.749769    2581 generic.go:356] "Generic (PLEG): container finished" podID="4d92766b-be44-4cc0-ae33-6d8574f4f604" containerID="ad5ed06edf4e499486aea915bb0a611dc3d97bf514a8c61c19a621f7f7ebd6df" exitCode=137
Sep 11 18:44:24.750267 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:24.749833    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7638-78e927"
Sep 11 18:44:24.750267 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:24.749882    2581 scope.go:117] "RemoveContainer" containerID="ad5ed06edf4e499486aea915bb0a611dc3d97bf514a8c61c19a621f7f7ebd6df"
Sep 11 18:44:24.752108 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:44:24.752079    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7638-78e927\" is forbidden: User \"system:node:ip-10-0-129-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-166.ec2.internal' and this object" podUID="4d92766b-be44-4cc0-ae33-6d8574f4f604" pod="opendatahub/test-curl-7638-78e927"
Sep 11 18:44:24.758792 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:24.758775    2581 scope.go:117] "RemoveContainer" containerID="ad5ed06edf4e499486aea915bb0a611dc3d97bf514a8c61c19a621f7f7ebd6df"
Sep 11 18:44:24.759061 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:44:24.759044    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad5ed06edf4e499486aea915bb0a611dc3d97bf514a8c61c19a621f7f7ebd6df\": container with ID starting with ad5ed06edf4e499486aea915bb0a611dc3d97bf514a8c61c19a621f7f7ebd6df not found: ID does not exist" containerID="ad5ed06edf4e499486aea915bb0a611dc3d97bf514a8c61c19a621f7f7ebd6df"
Sep 11 18:44:24.759137 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:24.759069    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad5ed06edf4e499486aea915bb0a611dc3d97bf514a8c61c19a621f7f7ebd6df"} err="failed to get container status \"ad5ed06edf4e499486aea915bb0a611dc3d97bf514a8c61c19a621f7f7ebd6df\": rpc error: code = NotFound desc = could not find container \"ad5ed06edf4e499486aea915bb0a611dc3d97bf514a8c61c19a621f7f7ebd6df\": container with ID starting with ad5ed06edf4e499486aea915bb0a611dc3d97bf514a8c61c19a621f7f7ebd6df not found: ID does not exist"
Sep 11 18:44:24.759471 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:44:24.759449    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7638-78e927\" is forbidden: User \"system:node:ip-10-0-129-166.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-166.ec2.internal' and this object" podUID="4d92766b-be44-4cc0-ae33-6d8574f4f604" pod="opendatahub/test-curl-7638-78e927"
Sep 11 18:44:25.456393 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:25.456356    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:44:25.763880 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:25.763786    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d92766b-be44-4cc0-ae33-6d8574f4f604" path="/var/lib/kubelet/pods/4d92766b-be44-4cc0-ae33-6d8574f4f604/volumes"
Sep 11 18:44:35.540401 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:35.540366    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:44:37.354471 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.354407    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb" podUID="7803e616-73e7-4e43-a404-90e336546bbd" containerName="main" containerID="cri-o://355d68b2bb46e8feed1c339943631530809579814a37a32613b660aeed682a01" gracePeriod=45
Sep 11 18:44:37.608678 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.608626    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:44:37.692673 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.692643    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz797\" (UniqueName: \"kubernetes.io/projected/7803e616-73e7-4e43-a404-90e336546bbd-kube-api-access-lz797\") pod \"7803e616-73e7-4e43-a404-90e336546bbd\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") "
Sep 11 18:44:37.692856 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.692705    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7803e616-73e7-4e43-a404-90e336546bbd-tls-certs\") pod \"7803e616-73e7-4e43-a404-90e336546bbd\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") "
Sep 11 18:44:37.692856 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.692747    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-tmp-dir\") pod \"7803e616-73e7-4e43-a404-90e336546bbd\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") "
Sep 11 18:44:37.692856 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.692778    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-model-cache\") pod \"7803e616-73e7-4e43-a404-90e336546bbd\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") "
Sep 11 18:44:37.692856 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.692821    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-dshm\") pod \"7803e616-73e7-4e43-a404-90e336546bbd\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") "
Sep 11 18:44:37.693089 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.692867    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-home\") pod \"7803e616-73e7-4e43-a404-90e336546bbd\" (UID: \"7803e616-73e7-4e43-a404-90e336546bbd\") "
Sep 11 18:44:37.693089 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.693023    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7803e616-73e7-4e43-a404-90e336546bbd" (UID: "7803e616-73e7-4e43-a404-90e336546bbd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:37.693089 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.693040    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-model-cache" (OuterVolumeSpecName: "model-cache") pod "7803e616-73e7-4e43-a404-90e336546bbd" (UID: "7803e616-73e7-4e43-a404-90e336546bbd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:37.693246 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.693193    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:44:37.693246 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.693213    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:44:37.693246 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.693218    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-home" (OuterVolumeSpecName: "home") pod "7803e616-73e7-4e43-a404-90e336546bbd" (UID: "7803e616-73e7-4e43-a404-90e336546bbd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:37.695072 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.695042    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7803e616-73e7-4e43-a404-90e336546bbd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7803e616-73e7-4e43-a404-90e336546bbd" (UID: "7803e616-73e7-4e43-a404-90e336546bbd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:44:37.695166 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.695066    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7803e616-73e7-4e43-a404-90e336546bbd-kube-api-access-lz797" (OuterVolumeSpecName: "kube-api-access-lz797") pod "7803e616-73e7-4e43-a404-90e336546bbd" (UID: "7803e616-73e7-4e43-a404-90e336546bbd"). InnerVolumeSpecName "kube-api-access-lz797". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:37.695166 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.695092    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-dshm" (OuterVolumeSpecName: "dshm") pod "7803e616-73e7-4e43-a404-90e336546bbd" (UID: "7803e616-73e7-4e43-a404-90e336546bbd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:37.794255 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.794229    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:44:37.794381 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.794258    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz797\" (UniqueName: \"kubernetes.io/projected/7803e616-73e7-4e43-a404-90e336546bbd-kube-api-access-lz797\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:44:37.794381 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.794273    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7803e616-73e7-4e43-a404-90e336546bbd-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:44:37.794381 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.794286    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7803e616-73e7-4e43-a404-90e336546bbd-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:44:37.813584 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.813554    2581 generic.go:356] "Generic (PLEG): container finished" podID="7803e616-73e7-4e43-a404-90e336546bbd" containerID="355d68b2bb46e8feed1c339943631530809579814a37a32613b660aeed682a01" exitCode=0
Sep 11 18:44:37.813688 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.813601    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"
Sep 11 18:44:37.813688 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.813645    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb" event={"ID":"7803e616-73e7-4e43-a404-90e336546bbd","Type":"ContainerDied","Data":"355d68b2bb46e8feed1c339943631530809579814a37a32613b660aeed682a01"}
Sep 11 18:44:37.813816 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.813690    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb" event={"ID":"7803e616-73e7-4e43-a404-90e336546bbd","Type":"ContainerDied","Data":"3660ffbafb8a25c52328f0c4025a3b0f31eced5bba699cf9cb1099850f0ee104"}
Sep 11 18:44:37.813816 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.813713    2581 scope.go:117] "RemoveContainer" containerID="355d68b2bb46e8feed1c339943631530809579814a37a32613b660aeed682a01"
Sep 11 18:44:37.822263 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.822246    2581 scope.go:117] "RemoveContainer" containerID="355d68b2bb46e8feed1c339943631530809579814a37a32613b660aeed682a01"
Sep 11 18:44:37.822505 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:44:37.822486    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355d68b2bb46e8feed1c339943631530809579814a37a32613b660aeed682a01\": container with ID starting with 355d68b2bb46e8feed1c339943631530809579814a37a32613b660aeed682a01 not found: ID does not exist" containerID="355d68b2bb46e8feed1c339943631530809579814a37a32613b660aeed682a01"
Sep 11 18:44:37.822585 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.822511    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355d68b2bb46e8feed1c339943631530809579814a37a32613b660aeed682a01"} err="failed to get container status \"355d68b2bb46e8feed1c339943631530809579814a37a32613b660aeed682a01\": rpc error: code = NotFound desc = could not find container \"355d68b2bb46e8feed1c339943631530809579814a37a32613b660aeed682a01\": container with ID starting with 355d68b2bb46e8feed1c339943631530809579814a37a32613b660aeed682a01 not found: ID does not exist"
Sep 11 18:44:37.830720 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.830701    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"]
Sep 11 18:44:37.834994 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:37.834976    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-7ca6df/sub-test-model-7ca6df-kserve-69d84bfcfc-qzlhb"]
Sep 11 18:44:38.306022 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.305985    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv" podUID="4920645c-66f7-4296-b236-c708ae4827df" containerName="main" containerID="cri-o://98b5f1788a2740d89769677c815d232c050b33bd2fb40eacc183cb4b925d41fb" gracePeriod=45
Sep 11 18:44:38.563528 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.563456    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:44:38.601376 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.601348    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-model-cache\") pod \"4920645c-66f7-4296-b236-c708ae4827df\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") "
Sep 11 18:44:38.601534 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.601385    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-dshm\") pod \"4920645c-66f7-4296-b236-c708ae4827df\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") "
Sep 11 18:44:38.601534 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.601451    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-home\") pod \"4920645c-66f7-4296-b236-c708ae4827df\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") "
Sep 11 18:44:38.601534 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.601508    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4920645c-66f7-4296-b236-c708ae4827df-tls-certs\") pod \"4920645c-66f7-4296-b236-c708ae4827df\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") "
Sep 11 18:44:38.601703 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.601535    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxt8r\" (UniqueName: \"kubernetes.io/projected/4920645c-66f7-4296-b236-c708ae4827df-kube-api-access-kxt8r\") pod \"4920645c-66f7-4296-b236-c708ae4827df\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") "
Sep 11 18:44:38.601703 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.601562    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-tmp-dir\") pod \"4920645c-66f7-4296-b236-c708ae4827df\" (UID: \"4920645c-66f7-4296-b236-c708ae4827df\") "
Sep 11 18:44:38.601703 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.601598    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-model-cache" (OuterVolumeSpecName: "model-cache") pod "4920645c-66f7-4296-b236-c708ae4827df" (UID: "4920645c-66f7-4296-b236-c708ae4827df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:38.601858 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.601780    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-home" (OuterVolumeSpecName: "home") pod "4920645c-66f7-4296-b236-c708ae4827df" (UID: "4920645c-66f7-4296-b236-c708ae4827df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:38.601917 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.601888    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4920645c-66f7-4296-b236-c708ae4827df" (UID: "4920645c-66f7-4296-b236-c708ae4827df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:38.601990 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.601970    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:44:38.602048 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.601996    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:44:38.602048 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.602014    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:44:38.603738 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.603711    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-dshm" (OuterVolumeSpecName: "dshm") pod "4920645c-66f7-4296-b236-c708ae4827df" (UID: "4920645c-66f7-4296-b236-c708ae4827df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:38.603848 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.603773    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4920645c-66f7-4296-b236-c708ae4827df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4920645c-66f7-4296-b236-c708ae4827df" (UID: "4920645c-66f7-4296-b236-c708ae4827df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:44:38.603848 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.603805    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4920645c-66f7-4296-b236-c708ae4827df-kube-api-access-kxt8r" (OuterVolumeSpecName: "kube-api-access-kxt8r") pod "4920645c-66f7-4296-b236-c708ae4827df" (UID: "4920645c-66f7-4296-b236-c708ae4827df"). InnerVolumeSpecName "kube-api-access-kxt8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:38.703323 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.703295    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxt8r\" (UniqueName: \"kubernetes.io/projected/4920645c-66f7-4296-b236-c708ae4827df-kube-api-access-kxt8r\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:44:38.703323 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.703319    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4920645c-66f7-4296-b236-c708ae4827df-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:44:38.703323 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.703329    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4920645c-66f7-4296-b236-c708ae4827df-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:44:38.819867 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.819796    2581 generic.go:356] "Generic (PLEG): container finished" podID="4920645c-66f7-4296-b236-c708ae4827df" containerID="98b5f1788a2740d89769677c815d232c050b33bd2fb40eacc183cb4b925d41fb" exitCode=0
Sep 11 18:44:38.819867 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.819830    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"
Sep 11 18:44:38.820051 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.819875    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv" event={"ID":"4920645c-66f7-4296-b236-c708ae4827df","Type":"ContainerDied","Data":"98b5f1788a2740d89769677c815d232c050b33bd2fb40eacc183cb4b925d41fb"}
Sep 11 18:44:38.820051 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.819917    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv" event={"ID":"4920645c-66f7-4296-b236-c708ae4827df","Type":"ContainerDied","Data":"14f971ab00183b63420173223f4ab6d3b2188c9702d8dc6ef65a599c9f888e8e"}
Sep 11 18:44:38.820051 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.819938    2581 scope.go:117] "RemoveContainer" containerID="98b5f1788a2740d89769677c815d232c050b33bd2fb40eacc183cb4b925d41fb"
Sep 11 18:44:38.828857 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.828841    2581 scope.go:117] "RemoveContainer" containerID="98b5f1788a2740d89769677c815d232c050b33bd2fb40eacc183cb4b925d41fb"
Sep 11 18:44:38.829117 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:44:38.829100    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b5f1788a2740d89769677c815d232c050b33bd2fb40eacc183cb4b925d41fb\": container with ID starting with 98b5f1788a2740d89769677c815d232c050b33bd2fb40eacc183cb4b925d41fb not found: ID does not exist" containerID="98b5f1788a2740d89769677c815d232c050b33bd2fb40eacc183cb4b925d41fb"
Sep 11 18:44:38.829169 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.829123    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b5f1788a2740d89769677c815d232c050b33bd2fb40eacc183cb4b925d41fb"} err="failed to get container status \"98b5f1788a2740d89769677c815d232c050b33bd2fb40eacc183cb4b925d41fb\": rpc error: code = NotFound desc = could not find container \"98b5f1788a2740d89769677c815d232c050b33bd2fb40eacc183cb4b925d41fb\": container with ID starting with 98b5f1788a2740d89769677c815d232c050b33bd2fb40eacc183cb4b925d41fb not found: ID does not exist"
Sep 11 18:44:38.841748 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.841727    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"]
Sep 11 18:44:38.846749 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:38.846727    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/sub-test-model-711a34-kserve-5f4c49ffb5-bdrwv"]
Sep 11 18:44:39.764317 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:39.764283    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4920645c-66f7-4296-b236-c708ae4827df" path="/var/lib/kubelet/pods/4920645c-66f7-4296-b236-c708ae4827df/volumes"
Sep 11 18:44:39.764716 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:39.764700    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7803e616-73e7-4e43-a404-90e336546bbd" path="/var/lib/kubelet/pods/7803e616-73e7-4e43-a404-90e336546bbd/volumes"
Sep 11 18:44:43.942603 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:43.942565    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:44:53.152216 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.152183    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"]
Sep 11 18:44:53.152681 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.152594    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7803e616-73e7-4e43-a404-90e336546bbd" containerName="main"
Sep 11 18:44:53.152681 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.152606    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7803e616-73e7-4e43-a404-90e336546bbd" containerName="main"
Sep 11 18:44:53.152681 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.152621    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4920645c-66f7-4296-b236-c708ae4827df" containerName="main"
Sep 11 18:44:53.152681 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.152626    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4920645c-66f7-4296-b236-c708ae4827df" containerName="main"
Sep 11 18:44:53.152681 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.152637    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d92766b-be44-4cc0-ae33-6d8574f4f604" containerName="test-curl-7638-78e927"
Sep 11 18:44:53.152681 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.152642    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d92766b-be44-4cc0-ae33-6d8574f4f604" containerName="test-curl-7638-78e927"
Sep 11 18:44:53.152877 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.152705    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7803e616-73e7-4e43-a404-90e336546bbd" containerName="main"
Sep 11 18:44:53.152877 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.152711    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4920645c-66f7-4296-b236-c708ae4827df" containerName="main"
Sep 11 18:44:53.152877 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.152718    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d92766b-be44-4cc0-ae33-6d8574f4f604" containerName="test-curl-7638-78e927"
Sep 11 18:44:53.156754 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.156736    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.159995 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.159970    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-711a34\"/\"rate-test-model-711a34-kserve-self-signed-certs\""
Sep 11 18:44:53.159995 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.159990    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-711a34\"/\"openshift-service-ca.crt\""
Sep 11 18:44:53.160188 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.160051    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-711a34\"/\"kube-root-ca.crt\""
Sep 11 18:44:53.160188 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.160062    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-711a34\"/\"default-dockercfg-f7z6v\""
Sep 11 18:44:53.166037 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.166014    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"]
Sep 11 18:44:53.242428 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.242404    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-dshm\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.242583 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.242439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-tmp-dir\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.242583 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.242571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-home\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.242681 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.242607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw6bd\" (UniqueName: \"kubernetes.io/projected/3198237c-b9b2-4632-94a4-c8bab89b1511-kube-api-access-fw6bd\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.242681 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.242658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3198237c-b9b2-4632-94a4-c8bab89b1511-tls-certs\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.242681 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.242678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-model-cache\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.343232 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.343204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-home\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.343378 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.343241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw6bd\" (UniqueName: \"kubernetes.io/projected/3198237c-b9b2-4632-94a4-c8bab89b1511-kube-api-access-fw6bd\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.343378 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.343268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3198237c-b9b2-4632-94a4-c8bab89b1511-tls-certs\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.343378 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.343282    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-model-cache\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.343378 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.343306    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-dshm\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.343378 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.343322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-tmp-dir\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.343676 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.343619    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-home\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.343676 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.343668    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-tmp-dir\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.343820 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.343720    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-model-cache\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.345709 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.345690    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-dshm\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.345959 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.345944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3198237c-b9b2-4632-94a4-c8bab89b1511-tls-certs\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.352293 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.352274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw6bd\" (UniqueName: \"kubernetes.io/projected/3198237c-b9b2-4632-94a4-c8bab89b1511-kube-api-access-fw6bd\") pod \"rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") " pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.466896 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.466823    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:44:53.740508 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.740454    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"]
Sep 11 18:44:53.742781 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:44:53.742753    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3198237c_b9b2_4632_94a4_c8bab89b1511.slice/crio-11ff8137e3f2eff53a303ad15c4edd677c4b754c178b65e348425f899c859bf8 WatchSource:0}: Error finding container 11ff8137e3f2eff53a303ad15c4edd677c4b754c178b65e348425f899c859bf8: Status 404 returned error can't find the container with id 11ff8137e3f2eff53a303ad15c4edd677c4b754c178b65e348425f899c859bf8
Sep 11 18:44:53.900997 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.900957    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr" event={"ID":"3198237c-b9b2-4632-94a4-c8bab89b1511","Type":"ContainerStarted","Data":"4c3fc8426be35f77ef7bf904b9d95b503cb4b87e8caecafeb01bd07a4dfa2d3b"}
Sep 11 18:44:53.901162 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.901012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr" event={"ID":"3198237c-b9b2-4632-94a4-c8bab89b1511","Type":"ContainerStarted","Data":"11ff8137e3f2eff53a303ad15c4edd677c4b754c178b65e348425f899c859bf8"}
Sep 11 18:44:53.925238 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:53.925195    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr" podStartSLOduration=0.925179199 podStartE2EDuration="925.179199ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:44:53.92337783 +0000 UTC m=+1770.664504673" watchObservedRunningTime="2026-09-11 18:44:53.925179199 +0000 UTC m=+1770.666306092"
Sep 11 18:44:57.049674 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:44:57.049634    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:45:00.132650 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.132612    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7ca6df-29819205-fr52q"]
Sep 11 18:45:00.136267 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.136246    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7ca6df-29819205-fr52q"
Sep 11 18:45:00.137852 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.137826    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819205-p5rcm"]
Sep 11 18:45:00.139039 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.139022    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\""
Sep 11 18:45:00.141324 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.141306    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-p5rcm"
Sep 11 18:45:00.144142 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.144123    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7ca6df-29819205-fr52q"]
Sep 11 18:45:00.149211 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.149186    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819205-p5rcm"]
Sep 11 18:45:00.149426 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.149402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7ca6df-29819205-fr52q"
Sep 11 18:45:00.159264 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.159245    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-p5rcm"
Sep 11 18:45:00.427092 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.427069    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7ca6df-29819205-fr52q"]
Sep 11 18:45:00.429087 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:45:00.429061    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ddbddb1_f3bf_4fb3_b20f_d4360d745c55.slice/crio-12373ff8ca441c1196bd4112639126b765fe49985b9b4772af4fd4a5cb33b388 WatchSource:0}: Error finding container 12373ff8ca441c1196bd4112639126b765fe49985b9b4772af4fd4a5cb33b388: Status 404 returned error can't find the container with id 12373ff8ca441c1196bd4112639126b765fe49985b9b4772af4fd4a5cb33b388
Sep 11 18:45:00.437108 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.436930    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819205-p5rcm"]
Sep 11 18:45:00.439079 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:45:00.439052    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154a5d37_e072_4f4c_bf3d_4f5c8253d753.slice/crio-31db8dfebcbb54e8bad6d3d2263297b66186f24890a23fd06405a66516ac86cc WatchSource:0}: Error finding container 31db8dfebcbb54e8bad6d3d2263297b66186f24890a23fd06405a66516ac86cc: Status 404 returned error can't find the container with id 31db8dfebcbb54e8bad6d3d2263297b66186f24890a23fd06405a66516ac86cc
Sep 11 18:45:00.941443 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.941405    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7ca6df-29819205-fr52q" event={"ID":"7ddbddb1-f3bf-4fb3-b20f-d4360d745c55","Type":"ContainerStarted","Data":"140f2790ad31ff3043105bd746d000588ef75de5e4c2f75c6c812d34b7d8fa12"}
Sep 11 18:45:00.941676 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.941470    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7ca6df-29819205-fr52q" event={"ID":"7ddbddb1-f3bf-4fb3-b20f-d4360d745c55","Type":"ContainerStarted","Data":"12373ff8ca441c1196bd4112639126b765fe49985b9b4772af4fd4a5cb33b388"}
Sep 11 18:45:00.943086 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.943059    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-p5rcm" event={"ID":"154a5d37-e072-4f4c-bf3d-4f5c8253d753","Type":"ContainerStarted","Data":"9fe8e2a900a5f590569d0de975501ae6b21c9b3e631e97e805c1dafcbed9674f"}
Sep 11 18:45:00.943205 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.943105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-p5rcm" event={"ID":"154a5d37-e072-4f4c-bf3d-4f5c8253d753","Type":"ContainerStarted","Data":"31db8dfebcbb54e8bad6d3d2263297b66186f24890a23fd06405a66516ac86cc"}
Sep 11 18:45:00.956695 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.956655    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7ca6df-29819205-fr52q" podStartSLOduration=0.956640625 podStartE2EDuration="956.640625ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:45:00.955760723 +0000 UTC m=+1777.696887567" watchObservedRunningTime="2026-09-11 18:45:00.956640625 +0000 UTC m=+1777.697767528"
Sep 11 18:45:00.970725 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:00.970679    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-p5rcm" podStartSLOduration=0.970664963 podStartE2EDuration="970.664963ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:45:00.969293398 +0000 UTC m=+1777.710420241" watchObservedRunningTime="2026-09-11 18:45:00.970664963 +0000 UTC m=+1777.711791811"
Sep 11 18:45:01.949876 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:01.949841    2581 generic.go:356] "Generic (PLEG): container finished" podID="154a5d37-e072-4f4c-bf3d-4f5c8253d753" containerID="9fe8e2a900a5f590569d0de975501ae6b21c9b3e631e97e805c1dafcbed9674f" exitCode=0
Sep 11 18:45:01.950299 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:01.949922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-p5rcm" event={"ID":"154a5d37-e072-4f4c-bf3d-4f5c8253d753","Type":"ContainerDied","Data":"9fe8e2a900a5f590569d0de975501ae6b21c9b3e631e97e805c1dafcbed9674f"}
Sep 11 18:45:01.951571 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:01.951552    2581 generic.go:356] "Generic (PLEG): container finished" podID="7ddbddb1-f3bf-4fb3-b20f-d4360d745c55" containerID="140f2790ad31ff3043105bd746d000588ef75de5e4c2f75c6c812d34b7d8fa12" exitCode=0
Sep 11 18:45:01.951714 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:01.951591    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7ca6df-29819205-fr52q" event={"ID":"7ddbddb1-f3bf-4fb3-b20f-d4360d745c55","Type":"ContainerDied","Data":"140f2790ad31ff3043105bd746d000588ef75de5e4c2f75c6c812d34b7d8fa12"}
Sep 11 18:45:03.091623 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:03.091593    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-p5rcm"
Sep 11 18:45:03.117231 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:03.117210    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7ca6df-29819205-fr52q"
Sep 11 18:45:03.467662 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:03.467628    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:45:03.467824 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:03.467684    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:45:03.480062 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:03.480037    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:45:03.962525 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:03.962470    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-p5rcm"
Sep 11 18:45:03.962525 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:03.962502    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-p5rcm" event={"ID":"154a5d37-e072-4f4c-bf3d-4f5c8253d753","Type":"ContainerDied","Data":"31db8dfebcbb54e8bad6d3d2263297b66186f24890a23fd06405a66516ac86cc"}
Sep 11 18:45:03.962771 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:03.962553    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31db8dfebcbb54e8bad6d3d2263297b66186f24890a23fd06405a66516ac86cc"
Sep 11 18:45:03.964016 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:03.963983    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7ca6df-29819205-fr52q" event={"ID":"7ddbddb1-f3bf-4fb3-b20f-d4360d745c55","Type":"ContainerDied","Data":"12373ff8ca441c1196bd4112639126b765fe49985b9b4772af4fd4a5cb33b388"}
Sep 11 18:45:03.964149 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:03.964037    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12373ff8ca441c1196bd4112639126b765fe49985b9b4772af4fd4a5cb33b388"
Sep 11 18:45:03.964149 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:03.964000    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7ca6df-29819205-fr52q"
Sep 11 18:45:03.975771 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:03.975750    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:45:20.744695 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:20.744620    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:45:30.039207 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:30.039165    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:45:37.847518 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:37.847460    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"]
Sep 11 18:45:40.841624 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:40.841586    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:45:44.572816 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.572781    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"]
Sep 11 18:45:44.573174 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.573161    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ddbddb1-f3bf-4fb3-b20f-d4360d745c55" containerName="cleanup"
Sep 11 18:45:44.573217 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.573175    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ddbddb1-f3bf-4fb3-b20f-d4360d745c55" containerName="cleanup"
Sep 11 18:45:44.573217 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.573185    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="154a5d37-e072-4f4c-bf3d-4f5c8253d753" containerName="cleanup"
Sep 11 18:45:44.573217 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.573190    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="154a5d37-e072-4f4c-bf3d-4f5c8253d753" containerName="cleanup"
Sep 11 18:45:44.573315 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.573257    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="154a5d37-e072-4f4c-bf3d-4f5c8253d753" containerName="cleanup"
Sep 11 18:45:44.573315 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.573265    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ddbddb1-f3bf-4fb3-b20f-d4360d745c55" containerName="cleanup"
Sep 11 18:45:44.577377 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.577360    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"
Sep 11 18:45:44.579327 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.579307    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b0293c\""
Sep 11 18:45:44.579428 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.579383    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b0293c-dockercfg-jxf8t\""
Sep 11 18:45:44.586517 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.586493    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"]
Sep 11 18:45:44.603963 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.603940    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"]
Sep 11 18:45:44.607265 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.607249    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"
Sep 11 18:45:44.618899 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.618877    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"]
Sep 11 18:45:44.714303 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.714272    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1c695d-598b-42ad-804d-37d9f01bb909-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64\" (UID: \"fd1c695d-598b-42ad-804d-37d9f01bb909\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"
Sep 11 18:45:44.714472 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.714332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ffb3c02-04d5-4ceb-9988-08f4e9892b89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk\" (UID: \"7ffb3c02-04d5-4ceb-9988-08f4e9892b89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"
Sep 11 18:45:44.714472 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.714385    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw72b\" (UniqueName: \"kubernetes.io/projected/fd1c695d-598b-42ad-804d-37d9f01bb909-kube-api-access-dw72b\") pod \"payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64\" (UID: \"fd1c695d-598b-42ad-804d-37d9f01bb909\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"
Sep 11 18:45:44.714579 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.714467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nttdp\" (UniqueName: \"kubernetes.io/projected/7ffb3c02-04d5-4ceb-9988-08f4e9892b89-kube-api-access-nttdp\") pod \"payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk\" (UID: \"7ffb3c02-04d5-4ceb-9988-08f4e9892b89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"
Sep 11 18:45:44.815743 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.815704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nttdp\" (UniqueName: \"kubernetes.io/projected/7ffb3c02-04d5-4ceb-9988-08f4e9892b89-kube-api-access-nttdp\") pod \"payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk\" (UID: \"7ffb3c02-04d5-4ceb-9988-08f4e9892b89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"
Sep 11 18:45:44.815923 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.815751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1c695d-598b-42ad-804d-37d9f01bb909-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64\" (UID: \"fd1c695d-598b-42ad-804d-37d9f01bb909\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"
Sep 11 18:45:44.815923 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.815793    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ffb3c02-04d5-4ceb-9988-08f4e9892b89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk\" (UID: \"7ffb3c02-04d5-4ceb-9988-08f4e9892b89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"
Sep 11 18:45:44.816036 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.815968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw72b\" (UniqueName: \"kubernetes.io/projected/fd1c695d-598b-42ad-804d-37d9f01bb909-kube-api-access-dw72b\") pod \"payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64\" (UID: \"fd1c695d-598b-42ad-804d-37d9f01bb909\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"
Sep 11 18:45:44.816450 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.816430    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1c695d-598b-42ad-804d-37d9f01bb909-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64\" (UID: \"fd1c695d-598b-42ad-804d-37d9f01bb909\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"
Sep 11 18:45:44.816551 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.816532    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ffb3c02-04d5-4ceb-9988-08f4e9892b89-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk\" (UID: \"7ffb3c02-04d5-4ceb-9988-08f4e9892b89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"
Sep 11 18:45:44.824099 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.824026    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw72b\" (UniqueName: \"kubernetes.io/projected/fd1c695d-598b-42ad-804d-37d9f01bb909-kube-api-access-dw72b\") pod \"payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64\" (UID: \"fd1c695d-598b-42ad-804d-37d9f01bb909\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"
Sep 11 18:45:44.824099 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.824028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nttdp\" (UniqueName: \"kubernetes.io/projected/7ffb3c02-04d5-4ceb-9988-08f4e9892b89-kube-api-access-nttdp\") pod \"payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk\" (UID: \"7ffb3c02-04d5-4ceb-9988-08f4e9892b89\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"
Sep 11 18:45:44.887924 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.887889    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"
Sep 11 18:45:44.916991 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:44.916956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"
Sep 11 18:45:45.167061 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:45.167035    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"]
Sep 11 18:45:45.168888 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:45:45.168854    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1c695d_598b_42ad_804d_37d9f01bb909.slice/crio-45b173064fc9646063a7ce26f210498f2cc843036e22719b4a77a785923ee294 WatchSource:0}: Error finding container 45b173064fc9646063a7ce26f210498f2cc843036e22719b4a77a785923ee294: Status 404 returned error can't find the container with id 45b173064fc9646063a7ce26f210498f2cc843036e22719b4a77a785923ee294
Sep 11 18:45:45.190257 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:45.190233    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64" event={"ID":"fd1c695d-598b-42ad-804d-37d9f01bb909","Type":"ContainerStarted","Data":"45b173064fc9646063a7ce26f210498f2cc843036e22719b4a77a785923ee294"}
Sep 11 18:45:45.194941 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:45.194919    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"]
Sep 11 18:45:45.200685 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:45:45.200662    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ffb3c02_04d5_4ceb_9988_08f4e9892b89.slice/crio-687ad6cd6fd78b8e41aa9284837320ed6d2f467c2c85cdffedd63163ffd5105c WatchSource:0}: Error finding container 687ad6cd6fd78b8e41aa9284837320ed6d2f467c2c85cdffedd63163ffd5105c: Status 404 returned error can't find the container with id 687ad6cd6fd78b8e41aa9284837320ed6d2f467c2c85cdffedd63163ffd5105c
Sep 11 18:45:46.197875 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:46.197841    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk" event={"ID":"7ffb3c02-04d5-4ceb-9988-08f4e9892b89","Type":"ContainerStarted","Data":"c042bcf72017368b1c4a45d2a621fae4226189d573b5a32efa96b07a7d77057b"}
Sep 11 18:45:46.198343 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:46.197901    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"
Sep 11 18:45:46.198343 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:46.197926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk" event={"ID":"7ffb3c02-04d5-4ceb-9988-08f4e9892b89","Type":"ContainerStarted","Data":"687ad6cd6fd78b8e41aa9284837320ed6d2f467c2c85cdffedd63163ffd5105c"}
Sep 11 18:45:46.199427 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:46.199396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64" event={"ID":"fd1c695d-598b-42ad-804d-37d9f01bb909","Type":"ContainerStarted","Data":"0f22d8d3a8d2a7e5898e030ccedb313c31243075935b8c822a0f2cf3b444c984"}
Sep 11 18:45:46.199557 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:46.199522    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"
Sep 11 18:45:46.216422 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:46.216381    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk" podStartSLOduration=2.2163674110000002 podStartE2EDuration="2.216367411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:45:46.214534156 +0000 UTC m=+1822.955661000" watchObservedRunningTime="2026-09-11 18:45:46.216367411 +0000 UTC m=+1822.957494281"
Sep 11 18:45:46.230157 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:46.230115    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64" podStartSLOduration=2.230102434 podStartE2EDuration="2.230102434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:45:46.229215733 +0000 UTC m=+1822.970342599" watchObservedRunningTime="2026-09-11 18:45:46.230102434 +0000 UTC m=+1822.971229277"
Sep 11 18:45:52.867806 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:52.867735    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr" podUID="3198237c-b9b2-4632-94a4-c8bab89b1511" containerName="main" containerID="cri-o://4c3fc8426be35f77ef7bf904b9d95b503cb4b87e8caecafeb01bd07a4dfa2d3b" gracePeriod=45
Sep 11 18:45:53.121903 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.121846    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:45:53.236942 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.236908    2581 generic.go:356] "Generic (PLEG): container finished" podID="3198237c-b9b2-4632-94a4-c8bab89b1511" containerID="4c3fc8426be35f77ef7bf904b9d95b503cb4b87e8caecafeb01bd07a4dfa2d3b" exitCode=0
Sep 11 18:45:53.237113 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.236950    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"
Sep 11 18:45:53.237113 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.236990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr" event={"ID":"3198237c-b9b2-4632-94a4-c8bab89b1511","Type":"ContainerDied","Data":"4c3fc8426be35f77ef7bf904b9d95b503cb4b87e8caecafeb01bd07a4dfa2d3b"}
Sep 11 18:45:53.237113 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.237038    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr" event={"ID":"3198237c-b9b2-4632-94a4-c8bab89b1511","Type":"ContainerDied","Data":"11ff8137e3f2eff53a303ad15c4edd677c4b754c178b65e348425f899c859bf8"}
Sep 11 18:45:53.237113 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.237060    2581 scope.go:117] "RemoveContainer" containerID="4c3fc8426be35f77ef7bf904b9d95b503cb4b87e8caecafeb01bd07a4dfa2d3b"
Sep 11 18:45:53.245895 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.245873    2581 scope.go:117] "RemoveContainer" containerID="4c3fc8426be35f77ef7bf904b9d95b503cb4b87e8caecafeb01bd07a4dfa2d3b"
Sep 11 18:45:53.246196 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:45:53.246173    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c3fc8426be35f77ef7bf904b9d95b503cb4b87e8caecafeb01bd07a4dfa2d3b\": container with ID starting with 4c3fc8426be35f77ef7bf904b9d95b503cb4b87e8caecafeb01bd07a4dfa2d3b not found: ID does not exist" containerID="4c3fc8426be35f77ef7bf904b9d95b503cb4b87e8caecafeb01bd07a4dfa2d3b"
Sep 11 18:45:53.246255 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.246208    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c3fc8426be35f77ef7bf904b9d95b503cb4b87e8caecafeb01bd07a4dfa2d3b"} err="failed to get container status \"4c3fc8426be35f77ef7bf904b9d95b503cb4b87e8caecafeb01bd07a4dfa2d3b\": rpc error: code = NotFound desc = could not find container \"4c3fc8426be35f77ef7bf904b9d95b503cb4b87e8caecafeb01bd07a4dfa2d3b\": container with ID starting with 4c3fc8426be35f77ef7bf904b9d95b503cb4b87e8caecafeb01bd07a4dfa2d3b not found: ID does not exist"
Sep 11 18:45:53.299387 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.299350    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-dshm\") pod \"3198237c-b9b2-4632-94a4-c8bab89b1511\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") "
Sep 11 18:45:53.299579 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.299411    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw6bd\" (UniqueName: \"kubernetes.io/projected/3198237c-b9b2-4632-94a4-c8bab89b1511-kube-api-access-fw6bd\") pod \"3198237c-b9b2-4632-94a4-c8bab89b1511\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") "
Sep 11 18:45:53.299579 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.299525    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3198237c-b9b2-4632-94a4-c8bab89b1511-tls-certs\") pod \"3198237c-b9b2-4632-94a4-c8bab89b1511\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") "
Sep 11 18:45:53.299579 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.299549    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-model-cache\") pod \"3198237c-b9b2-4632-94a4-c8bab89b1511\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") "
Sep 11 18:45:53.299579 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.299566    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-tmp-dir\") pod \"3198237c-b9b2-4632-94a4-c8bab89b1511\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") "
Sep 11 18:45:53.299793 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.299629    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-home\") pod \"3198237c-b9b2-4632-94a4-c8bab89b1511\" (UID: \"3198237c-b9b2-4632-94a4-c8bab89b1511\") "
Sep 11 18:45:53.299859 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.299837    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-model-cache" (OuterVolumeSpecName: "model-cache") pod "3198237c-b9b2-4632-94a4-c8bab89b1511" (UID: "3198237c-b9b2-4632-94a4-c8bab89b1511"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:45:53.299972 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.299953    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:45:53.299972 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.299957    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3198237c-b9b2-4632-94a4-c8bab89b1511" (UID: "3198237c-b9b2-4632-94a4-c8bab89b1511"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:45:53.300136 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.300027    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-home" (OuterVolumeSpecName: "home") pod "3198237c-b9b2-4632-94a4-c8bab89b1511" (UID: "3198237c-b9b2-4632-94a4-c8bab89b1511"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:45:53.301775 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.301752    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-dshm" (OuterVolumeSpecName: "dshm") pod "3198237c-b9b2-4632-94a4-c8bab89b1511" (UID: "3198237c-b9b2-4632-94a4-c8bab89b1511"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:45:53.301775 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.301766    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3198237c-b9b2-4632-94a4-c8bab89b1511-kube-api-access-fw6bd" (OuterVolumeSpecName: "kube-api-access-fw6bd") pod "3198237c-b9b2-4632-94a4-c8bab89b1511" (UID: "3198237c-b9b2-4632-94a4-c8bab89b1511"). InnerVolumeSpecName "kube-api-access-fw6bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:45:53.301913 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.301791    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3198237c-b9b2-4632-94a4-c8bab89b1511-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3198237c-b9b2-4632-94a4-c8bab89b1511" (UID: "3198237c-b9b2-4632-94a4-c8bab89b1511"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:45:53.401520 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.401466    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3198237c-b9b2-4632-94a4-c8bab89b1511-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:45:53.401680 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.401536    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:45:53.401680 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.401544    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:45:53.401680 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.401551    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3198237c-b9b2-4632-94a4-c8bab89b1511-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:45:53.401680 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.401562    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw6bd\" (UniqueName: \"kubernetes.io/projected/3198237c-b9b2-4632-94a4-c8bab89b1511-kube-api-access-fw6bd\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:45:53.559239 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.559213    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"]
Sep 11 18:45:53.564522 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.564472    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-711a34/rate-test-model-711a34-kserve-79cf7c7cf9-hmsjr"]
Sep 11 18:45:53.764810 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:53.764736    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3198237c-b9b2-4632-94a4-c8bab89b1511" path="/var/lib/kubelet/pods/3198237c-b9b2-4632-94a4-c8bab89b1511/volumes"
Sep 11 18:45:54.536957 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:54.536919    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:45:57.206542 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:57.206503    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"
Sep 11 18:45:57.206993 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:45:57.206663    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"
Sep 11 18:46:14.242057 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.242026    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"]
Sep 11 18:46:14.244504 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.242439    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3198237c-b9b2-4632-94a4-c8bab89b1511" containerName="main"
Sep 11 18:46:14.244504 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.242449    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3198237c-b9b2-4632-94a4-c8bab89b1511" containerName="main"
Sep 11 18:46:14.244504 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.242534    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3198237c-b9b2-4632-94a4-c8bab89b1511" containerName="main"
Sep 11 18:46:14.245683 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.245668    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"
Sep 11 18:46:14.247598 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.247571    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-946072-dockercfg-hccw4\""
Sep 11 18:46:14.247598 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.247571    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-946072\""
Sep 11 18:46:14.256845 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.256821    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"]
Sep 11 18:46:14.269752 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.269719    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"]
Sep 11 18:46:14.273360 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.273340    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"
Sep 11 18:46:14.281933 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.281909    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"]
Sep 11 18:46:14.393664 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.393632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7\" (UID: \"7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"
Sep 11 18:46:14.393832 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.393702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/191cb6e8-9601-4a2d-9258-80b84f2ae9ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv\" (UID: \"191cb6e8-9601-4a2d-9258-80b84f2ae9ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"
Sep 11 18:46:14.393832 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.393792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l56pw\" (UniqueName: \"kubernetes.io/projected/191cb6e8-9601-4a2d-9258-80b84f2ae9ba-kube-api-access-l56pw\") pod \"payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv\" (UID: \"191cb6e8-9601-4a2d-9258-80b84f2ae9ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"
Sep 11 18:46:14.393832 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.393823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp8s8\" (UniqueName: \"kubernetes.io/projected/7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b-kube-api-access-tp8s8\") pod \"payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7\" (UID: \"7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"
Sep 11 18:46:14.494494 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.494400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/191cb6e8-9601-4a2d-9258-80b84f2ae9ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv\" (UID: \"191cb6e8-9601-4a2d-9258-80b84f2ae9ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"
Sep 11 18:46:14.494494 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.494441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l56pw\" (UniqueName: \"kubernetes.io/projected/191cb6e8-9601-4a2d-9258-80b84f2ae9ba-kube-api-access-l56pw\") pod \"payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv\" (UID: \"191cb6e8-9601-4a2d-9258-80b84f2ae9ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"
Sep 11 18:46:14.494494 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.494463    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp8s8\" (UniqueName: \"kubernetes.io/projected/7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b-kube-api-access-tp8s8\") pod \"payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7\" (UID: \"7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"
Sep 11 18:46:14.494767 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.494560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7\" (UID: \"7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"
Sep 11 18:46:14.495063 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.495043    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/191cb6e8-9601-4a2d-9258-80b84f2ae9ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv\" (UID: \"191cb6e8-9601-4a2d-9258-80b84f2ae9ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"
Sep 11 18:46:14.495244 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.495227    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7\" (UID: \"7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"
Sep 11 18:46:14.502987 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.502961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l56pw\" (UniqueName: \"kubernetes.io/projected/191cb6e8-9601-4a2d-9258-80b84f2ae9ba-kube-api-access-l56pw\") pod \"payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv\" (UID: \"191cb6e8-9601-4a2d-9258-80b84f2ae9ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"
Sep 11 18:46:14.503105 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.503055    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp8s8\" (UniqueName: \"kubernetes.io/projected/7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b-kube-api-access-tp8s8\") pod \"payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7\" (UID: \"7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"
Sep 11 18:46:14.557215 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.557191    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"
Sep 11 18:46:14.584256 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.584233    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"
Sep 11 18:46:14.832653 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.832630    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"]
Sep 11 18:46:14.833960 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:46:14.833932    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2384c0_db5e_4e9a_87d3_4bfa02cdde5b.slice/crio-1262ea5f21d92df0db68b840ce43c60b0c477d4e8c774f6b1c243475ef951eda WatchSource:0}: Error finding container 1262ea5f21d92df0db68b840ce43c60b0c477d4e8c774f6b1c243475ef951eda: Status 404 returned error can't find the container with id 1262ea5f21d92df0db68b840ce43c60b0c477d4e8c774f6b1c243475ef951eda
Sep 11 18:46:14.868009 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:14.867967    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"]
Sep 11 18:46:14.869626 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:46:14.869600    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod191cb6e8_9601_4a2d_9258_80b84f2ae9ba.slice/crio-155d795778f3b181d8d2910296db376cb2934a8470710b48763492043a55b195 WatchSource:0}: Error finding container 155d795778f3b181d8d2910296db376cb2934a8470710b48763492043a55b195: Status 404 returned error can't find the container with id 155d795778f3b181d8d2910296db376cb2934a8470710b48763492043a55b195
Sep 11 18:46:15.355884 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:15.355833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7" event={"ID":"7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b","Type":"ContainerStarted","Data":"6efb2562cc3c33354b501f41e1cd798c294871e931c114f1837b1a212b9c02a7"}
Sep 11 18:46:15.356340 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:15.355901    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"
Sep 11 18:46:15.356340 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:15.355927    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7" event={"ID":"7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b","Type":"ContainerStarted","Data":"1262ea5f21d92df0db68b840ce43c60b0c477d4e8c774f6b1c243475ef951eda"}
Sep 11 18:46:15.357544 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:15.357521    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv" event={"ID":"191cb6e8-9601-4a2d-9258-80b84f2ae9ba","Type":"ContainerStarted","Data":"60f094691af6fb70df96e1940ef12cfcbe5955542aeb5ecf3a2ec313afc8a09e"}
Sep 11 18:46:15.357630 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:15.357562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv" event={"ID":"191cb6e8-9601-4a2d-9258-80b84f2ae9ba","Type":"ContainerStarted","Data":"155d795778f3b181d8d2910296db376cb2934a8470710b48763492043a55b195"}
Sep 11 18:46:15.357630 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:15.357582    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"
Sep 11 18:46:15.372692 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:15.372645    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7" podStartSLOduration=1.372629744 podStartE2EDuration="1.372629744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:46:15.370811394 +0000 UTC m=+1852.111938239" watchObservedRunningTime="2026-09-11 18:46:15.372629744 +0000 UTC m=+1852.113756587"
Sep 11 18:46:15.388289 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:15.388235    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv" podStartSLOduration=1.388216618 podStartE2EDuration="1.388216618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:46:15.387077693 +0000 UTC m=+1852.128204560" watchObservedRunningTime="2026-09-11 18:46:15.388216618 +0000 UTC m=+1852.129343461"
Sep 11 18:46:18.848660 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:18.848627    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:46:26.364966 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:26.364934    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"
Sep 11 18:46:26.365465 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:26.365074    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"
Sep 11 18:46:41.075324 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:41.075288    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2292w_cc357e4e-df13-4296-84eb-8d0d846b34be/payload-processing/0.log"
Sep 11 18:46:41.193540 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:41.193510    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv_191cb6e8-9601-4a2d-9258-80b84f2ae9ba/payload-processing/0.log"
Sep 11 18:46:48.541690 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.541614    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"]
Sep 11 18:46:48.546176 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.546158    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.548609 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.548582    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b0293c\"/\"ipp-route-b0293c-kserve-self-signed-certs\""
Sep 11 18:46:48.549324 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.549308    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b0293c\"/\"kube-root-ca.crt\""
Sep 11 18:46:48.549412 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.549309    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b0293c\"/\"default-dockercfg-6vpl5\""
Sep 11 18:46:48.549412 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.549360    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b0293c\"/\"openshift-service-ca.crt\""
Sep 11 18:46:48.553584 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.553560    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"]
Sep 11 18:46:48.598786 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.598749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-tls-certs\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.598786 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.598784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-dshm\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.598987 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.598853    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-model-cache\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.598987 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.598874    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s99l8\" (UniqueName: \"kubernetes.io/projected/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-kube-api-access-s99l8\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.598987 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.598941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-tmp-dir\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.599088 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.598989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-home\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.699426 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.699388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-tmp-dir\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.699426 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.699428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-home\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.699770 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.699618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-tls-certs\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.699770 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.699651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-dshm\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.699770 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.699742    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-model-cache\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.699770 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.699754    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-home\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.699975 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.699744    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-tmp-dir\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.699975 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.699783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s99l8\" (UniqueName: \"kubernetes.io/projected/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-kube-api-access-s99l8\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.700071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.700009    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-model-cache\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.702173 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.702143    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-dshm\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.702369 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.702349    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-tls-certs\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.709283 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.709253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s99l8\" (UniqueName: \"kubernetes.io/projected/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-kube-api-access-s99l8\") pod \"ipp-route-b0293c-kserve-54fddf4587-pslkr\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") " pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:48.857099 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:48.857003    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:49.129253 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:49.129224    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"]
Sep 11 18:46:49.130663 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:46:49.130640    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02d77d70_0b38_4f7e_8011_aa53c61f0b1e.slice/crio-762312c839e9a9c5e7e59898055fb4f86bc76c71b2ecea1a7ce86798048d7833 WatchSource:0}: Error finding container 762312c839e9a9c5e7e59898055fb4f86bc76c71b2ecea1a7ce86798048d7833: Status 404 returned error can't find the container with id 762312c839e9a9c5e7e59898055fb4f86bc76c71b2ecea1a7ce86798048d7833
Sep 11 18:46:49.544342 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:49.544254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr" event={"ID":"02d77d70-0b38-4f7e-8011-aa53c61f0b1e","Type":"ContainerStarted","Data":"fe166473fe8ce6590417bffd12459f144e116030b16a82b94eb80b04a01c3d46"}
Sep 11 18:46:49.544342 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:49.544300    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr" event={"ID":"02d77d70-0b38-4f7e-8011-aa53c61f0b1e","Type":"ContainerStarted","Data":"762312c839e9a9c5e7e59898055fb4f86bc76c71b2ecea1a7ce86798048d7833"}
Sep 11 18:46:49.562677 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:49.562627    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr" podStartSLOduration=1.5626064579999999 podStartE2EDuration="1.562606458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:46:49.560739487 +0000 UTC m=+1886.301866332" watchObservedRunningTime="2026-09-11 18:46:49.562606458 +0000 UTC m=+1886.303733305"
Sep 11 18:46:52.743031 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:52.742994    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:46:58.857520 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:58.857464    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:58.857918 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:58.857531    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:58.870128 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:58.870103    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:46:59.615754 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:46:59.615718    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:47:06.044050 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:06.044014    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:47:11.837648 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:11.837616    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk_7ffb3c02-04d5-4ceb-9988-08f4e9892b89/payload-processing/0.log"
Sep 11 18:47:11.963261 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:11.963228    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2292w_cc357e4e-df13-4296-84eb-8d0d846b34be/payload-processing/0.log"
Sep 11 18:47:12.083000 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:12.082965    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv_191cb6e8-9601-4a2d-9258-80b84f2ae9ba/payload-processing/0.log"
Sep 11 18:47:20.031931 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.031901    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"]
Sep 11 18:47:20.042941 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.042907    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"]
Sep 11 18:47:20.043091 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.043023    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"
Sep 11 18:47:20.045162 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.045130    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-e9c09a-dockercfg-8gpqd\""
Sep 11 18:47:20.045350 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.045138    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-e9c09a\""
Sep 11 18:47:20.046469 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.046449    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"
Sep 11 18:47:20.049151 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.049129    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"]
Sep 11 18:47:20.053646 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.053625    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"]
Sep 11 18:47:20.201782 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.201741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d6cf98-da2d-4b02-ba01-c08cf2955f6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2\" (UID: \"e5d6cf98-da2d-4b02-ba01-c08cf2955f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"
Sep 11 18:47:20.202014 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.201791    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkn74\" (UniqueName: \"kubernetes.io/projected/e8b8e183-6a12-4a09-b65c-e1ff8299c760-kube-api-access-tkn74\") pod \"payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq\" (UID: \"e8b8e183-6a12-4a09-b65c-e1ff8299c760\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"
Sep 11 18:47:20.202014 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.201879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr27w\" (UniqueName: \"kubernetes.io/projected/e5d6cf98-da2d-4b02-ba01-c08cf2955f6f-kube-api-access-cr27w\") pod \"payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2\" (UID: \"e5d6cf98-da2d-4b02-ba01-c08cf2955f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"
Sep 11 18:47:20.202014 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.201929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b8e183-6a12-4a09-b65c-e1ff8299c760-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq\" (UID: \"e8b8e183-6a12-4a09-b65c-e1ff8299c760\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"
Sep 11 18:47:20.303540 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.303443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b8e183-6a12-4a09-b65c-e1ff8299c760-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq\" (UID: \"e8b8e183-6a12-4a09-b65c-e1ff8299c760\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"
Sep 11 18:47:20.303702 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.303543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d6cf98-da2d-4b02-ba01-c08cf2955f6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2\" (UID: \"e5d6cf98-da2d-4b02-ba01-c08cf2955f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"
Sep 11 18:47:20.303702 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.303571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkn74\" (UniqueName: \"kubernetes.io/projected/e8b8e183-6a12-4a09-b65c-e1ff8299c760-kube-api-access-tkn74\") pod \"payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq\" (UID: \"e8b8e183-6a12-4a09-b65c-e1ff8299c760\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"
Sep 11 18:47:20.303702 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.303614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr27w\" (UniqueName: \"kubernetes.io/projected/e5d6cf98-da2d-4b02-ba01-c08cf2955f6f-kube-api-access-cr27w\") pod \"payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2\" (UID: \"e5d6cf98-da2d-4b02-ba01-c08cf2955f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"
Sep 11 18:47:20.304133 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.304111    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b8e183-6a12-4a09-b65c-e1ff8299c760-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq\" (UID: \"e8b8e183-6a12-4a09-b65c-e1ff8299c760\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"
Sep 11 18:47:20.304244 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.304228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d6cf98-da2d-4b02-ba01-c08cf2955f6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2\" (UID: \"e5d6cf98-da2d-4b02-ba01-c08cf2955f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"
Sep 11 18:47:20.312579 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.312554    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkn74\" (UniqueName: \"kubernetes.io/projected/e8b8e183-6a12-4a09-b65c-e1ff8299c760-kube-api-access-tkn74\") pod \"payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq\" (UID: \"e8b8e183-6a12-4a09-b65c-e1ff8299c760\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"
Sep 11 18:47:20.312673 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.312650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr27w\" (UniqueName: \"kubernetes.io/projected/e5d6cf98-da2d-4b02-ba01-c08cf2955f6f-kube-api-access-cr27w\") pod \"payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2\" (UID: \"e5d6cf98-da2d-4b02-ba01-c08cf2955f6f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"
Sep 11 18:47:20.359657 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.359634    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"
Sep 11 18:47:20.366572 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.366555    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"
Sep 11 18:47:20.633364 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.633340    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"]
Sep 11 18:47:20.635366 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:47:20.635344    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d6cf98_da2d_4b02_ba01_c08cf2955f6f.slice/crio-da30c450cf6c321a91b1bb6ea2d505f5864167f554ba2a8288279b7277f00cfe WatchSource:0}: Error finding container da30c450cf6c321a91b1bb6ea2d505f5864167f554ba2a8288279b7277f00cfe: Status 404 returned error can't find the container with id da30c450cf6c321a91b1bb6ea2d505f5864167f554ba2a8288279b7277f00cfe
Sep 11 18:47:20.645903 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.645880    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"]
Sep 11 18:47:20.648897 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:47:20.648877    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b8e183_6a12_4a09_b65c_e1ff8299c760.slice/crio-da60e53c35656904e9e167a83005b12188c6b72c518ffbfbe6899136ea775fa9 WatchSource:0}: Error finding container da60e53c35656904e9e167a83005b12188c6b72c518ffbfbe6899136ea775fa9: Status 404 returned error can't find the container with id da60e53c35656904e9e167a83005b12188c6b72c518ffbfbe6899136ea775fa9
Sep 11 18:47:20.726655 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.726618    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq" event={"ID":"e8b8e183-6a12-4a09-b65c-e1ff8299c760","Type":"ContainerStarted","Data":"133aa6c05eeef51a0e140289b7057c654b879deb2bff8c433a1123964eaff361"}
Sep 11 18:47:20.726835 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.726682    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq" event={"ID":"e8b8e183-6a12-4a09-b65c-e1ff8299c760","Type":"ContainerStarted","Data":"da60e53c35656904e9e167a83005b12188c6b72c518ffbfbe6899136ea775fa9"}
Sep 11 18:47:20.726835 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.726801    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"
Sep 11 18:47:20.728304 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.728279    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2" event={"ID":"e5d6cf98-da2d-4b02-ba01-c08cf2955f6f","Type":"ContainerStarted","Data":"4b824fc2837f7daae07bc4855de30aa5f239b6d0a09020d642e3b769dc4cd1a7"}
Sep 11 18:47:20.728407 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.728317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2" event={"ID":"e5d6cf98-da2d-4b02-ba01-c08cf2955f6f","Type":"ContainerStarted","Data":"da30c450cf6c321a91b1bb6ea2d505f5864167f554ba2a8288279b7277f00cfe"}
Sep 11 18:47:20.728407 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.728399    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"
Sep 11 18:47:20.744169 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.744088    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq" podStartSLOduration=0.744070038 podStartE2EDuration="744.070038ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:47:20.741282533 +0000 UTC m=+1917.482409376" watchObservedRunningTime="2026-09-11 18:47:20.744070038 +0000 UTC m=+1917.485196881"
Sep 11 18:47:20.756746 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:20.756693    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2" podStartSLOduration=0.756679821 podStartE2EDuration="756.679821ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:47:20.755422502 +0000 UTC m=+1917.496549358" watchObservedRunningTime="2026-09-11 18:47:20.756679821 +0000 UTC m=+1917.497806688"
Sep 11 18:47:24.059466 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:24.059431    2581 scope.go:117] "RemoveContainer" containerID="9b95e3a367e6bd1a7cdd6092bca73a2a4efc46d8879f1af924948d645f67ad8c"
Sep 11 18:47:24.068234 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:24.068213    2581 scope.go:117] "RemoveContainer" containerID="25a1a0d932955197ef54a91f41ce8c6729a5cd207d374525912092733fed5b37"
Sep 11 18:47:25.253326 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:25.253295    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:47:31.737235 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:31.737201    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"
Sep 11 18:47:31.737750 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:31.737390    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"
Sep 11 18:47:46.373294 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:46.373262    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d"]
Sep 11 18:47:46.376808 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:46.376792    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d"
Sep 11 18:47:46.378712 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:46.378695    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\""
Sep 11 18:47:46.378712 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:46.378701    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:47:46.384200 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:46.384174    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d"]
Sep 11 18:47:46.441379 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:46.441343    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/890d0bc0-d8c6-4814-882c-4116ac0781bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d\" (UID: \"890d0bc0-d8c6-4814-882c-4116ac0781bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d"
Sep 11 18:47:46.542967 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:46.542931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/890d0bc0-d8c6-4814-882c-4116ac0781bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d\" (UID: \"890d0bc0-d8c6-4814-882c-4116ac0781bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d"
Sep 11 18:47:46.543533 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:46.543517    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/890d0bc0-d8c6-4814-882c-4116ac0781bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d\" (UID: \"890d0bc0-d8c6-4814-882c-4116ac0781bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d"
Sep 11 18:47:46.689096 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:46.689068    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d"
Sep 11 18:47:46.958766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:46.958742    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d"]
Sep 11 18:47:46.960921 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:47:46.960888    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890d0bc0_d8c6_4814_882c_4116ac0781bc.slice/crio-21e2a86a442122c08d315a81daa54387515fef6c217e00ec0753248e55c77257 WatchSource:0}: Error finding container 21e2a86a442122c08d315a81daa54387515fef6c217e00ec0753248e55c77257: Status 404 returned error can't find the container with id 21e2a86a442122c08d315a81daa54387515fef6c217e00ec0753248e55c77257
Sep 11 18:47:47.877750 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:47.877712    2581 generic.go:356] "Generic (PLEG): container finished" podID="890d0bc0-d8c6-4814-882c-4116ac0781bc" containerID="da90a63bd7974bf61013e6ffdfb585a5d5a7f26e944ecb43f6d50d900d2df270" exitCode=0
Sep 11 18:47:47.878095 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:47.877786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d" event={"ID":"890d0bc0-d8c6-4814-882c-4116ac0781bc","Type":"ContainerDied","Data":"da90a63bd7974bf61013e6ffdfb585a5d5a7f26e944ecb43f6d50d900d2df270"}
Sep 11 18:47:47.878095 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:47.877837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d" event={"ID":"890d0bc0-d8c6-4814-882c-4116ac0781bc","Type":"ContainerStarted","Data":"21e2a86a442122c08d315a81daa54387515fef6c217e00ec0753248e55c77257"}
Sep 11 18:47:49.018781 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:49.018757    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d"
Sep 11 18:47:49.065896 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:49.065863    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/890d0bc0-d8c6-4814-882c-4116ac0781bc-maas-api-service-ca\") pod \"890d0bc0-d8c6-4814-882c-4116ac0781bc\" (UID: \"890d0bc0-d8c6-4814-882c-4116ac0781bc\") "
Sep 11 18:47:49.066208 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:49.066183    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890d0bc0-d8c6-4814-882c-4116ac0781bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "890d0bc0-d8c6-4814-882c-4116ac0781bc" (UID: "890d0bc0-d8c6-4814-882c-4116ac0781bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:47:49.167129 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:49.167060    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/890d0bc0-d8c6-4814-882c-4116ac0781bc-maas-api-service-ca\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:47:49.889966 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:49.889938    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d"
Sep 11 18:47:49.889966 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:49.889948    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d" event={"ID":"890d0bc0-d8c6-4814-882c-4116ac0781bc","Type":"ContainerDied","Data":"21e2a86a442122c08d315a81daa54387515fef6c217e00ec0753248e55c77257"}
Sep 11 18:47:49.890160 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:49.889989    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21e2a86a442122c08d315a81daa54387515fef6c217e00ec0753248e55c77257"
Sep 11 18:47:57.535958 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.535926    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"]
Sep 11 18:47:57.536504 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.536153    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq" podUID="e8b8e183-6a12-4a09-b65c-e1ff8299c760" containerName="payload-processing" containerID="cri-o://133aa6c05eeef51a0e140289b7057c654b879deb2bff8c433a1123964eaff361" gracePeriod=30
Sep 11 18:47:57.541670 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.541632    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"]
Sep 11 18:47:57.541918 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.541870    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2" podUID="e5d6cf98-da2d-4b02-ba01-c08cf2955f6f" containerName="payload-pre-processing" containerID="cri-o://4b824fc2837f7daae07bc4855de30aa5f239b6d0a09020d642e3b769dc4cd1a7" gracePeriod=30
Sep 11 18:47:57.720125 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.720079    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d"]
Sep 11 18:47:57.722446 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.722421    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-e9c09a-ptg2d"]
Sep 11 18:47:57.765642 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.765615    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="890d0bc0-d8c6-4814-882c-4116ac0781bc" path="/var/lib/kubelet/pods/890d0bc0-d8c6-4814-882c-4116ac0781bc/volumes"
Sep 11 18:47:57.806111 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.806090    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"
Sep 11 18:47:57.831844 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.831819    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"
Sep 11 18:47:57.934078 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.934049    2581 generic.go:356] "Generic (PLEG): container finished" podID="e8b8e183-6a12-4a09-b65c-e1ff8299c760" containerID="133aa6c05eeef51a0e140289b7057c654b879deb2bff8c433a1123964eaff361" exitCode=0
Sep 11 18:47:57.934245 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.934091    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"
Sep 11 18:47:57.934245 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.934130    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq" event={"ID":"e8b8e183-6a12-4a09-b65c-e1ff8299c760","Type":"ContainerDied","Data":"133aa6c05eeef51a0e140289b7057c654b879deb2bff8c433a1123964eaff361"}
Sep 11 18:47:57.934245 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.934178    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq" event={"ID":"e8b8e183-6a12-4a09-b65c-e1ff8299c760","Type":"ContainerDied","Data":"da60e53c35656904e9e167a83005b12188c6b72c518ffbfbe6899136ea775fa9"}
Sep 11 18:47:57.934245 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.934201    2581 scope.go:117] "RemoveContainer" containerID="133aa6c05eeef51a0e140289b7057c654b879deb2bff8c433a1123964eaff361"
Sep 11 18:47:57.935335 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.935309    2581 generic.go:356] "Generic (PLEG): container finished" podID="e5d6cf98-da2d-4b02-ba01-c08cf2955f6f" containerID="4b824fc2837f7daae07bc4855de30aa5f239b6d0a09020d642e3b769dc4cd1a7" exitCode=0
Sep 11 18:47:57.935450 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.935350    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"
Sep 11 18:47:57.935450 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.935391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2" event={"ID":"e5d6cf98-da2d-4b02-ba01-c08cf2955f6f","Type":"ContainerDied","Data":"4b824fc2837f7daae07bc4855de30aa5f239b6d0a09020d642e3b769dc4cd1a7"}
Sep 11 18:47:57.935450 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.935445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2" event={"ID":"e5d6cf98-da2d-4b02-ba01-c08cf2955f6f","Type":"ContainerDied","Data":"da30c450cf6c321a91b1bb6ea2d505f5864167f554ba2a8288279b7277f00cfe"}
Sep 11 18:47:57.942832 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.942814    2581 scope.go:117] "RemoveContainer" containerID="133aa6c05eeef51a0e140289b7057c654b879deb2bff8c433a1123964eaff361"
Sep 11 18:47:57.943090 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:47:57.943072    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133aa6c05eeef51a0e140289b7057c654b879deb2bff8c433a1123964eaff361\": container with ID starting with 133aa6c05eeef51a0e140289b7057c654b879deb2bff8c433a1123964eaff361 not found: ID does not exist" containerID="133aa6c05eeef51a0e140289b7057c654b879deb2bff8c433a1123964eaff361"
Sep 11 18:47:57.943163 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.943096    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133aa6c05eeef51a0e140289b7057c654b879deb2bff8c433a1123964eaff361"} err="failed to get container status \"133aa6c05eeef51a0e140289b7057c654b879deb2bff8c433a1123964eaff361\": rpc error: code = NotFound desc = could not find container \"133aa6c05eeef51a0e140289b7057c654b879deb2bff8c433a1123964eaff361\": container with ID starting with 133aa6c05eeef51a0e140289b7057c654b879deb2bff8c433a1123964eaff361 not found: ID does not exist"
Sep 11 18:47:57.943163 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.943110    2581 scope.go:117] "RemoveContainer" containerID="4b824fc2837f7daae07bc4855de30aa5f239b6d0a09020d642e3b769dc4cd1a7"
Sep 11 18:47:57.950900 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.950882    2581 scope.go:117] "RemoveContainer" containerID="4b824fc2837f7daae07bc4855de30aa5f239b6d0a09020d642e3b769dc4cd1a7"
Sep 11 18:47:57.951168 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:47:57.951150    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b824fc2837f7daae07bc4855de30aa5f239b6d0a09020d642e3b769dc4cd1a7\": container with ID starting with 4b824fc2837f7daae07bc4855de30aa5f239b6d0a09020d642e3b769dc4cd1a7 not found: ID does not exist" containerID="4b824fc2837f7daae07bc4855de30aa5f239b6d0a09020d642e3b769dc4cd1a7"
Sep 11 18:47:57.951219 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.951175    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b824fc2837f7daae07bc4855de30aa5f239b6d0a09020d642e3b769dc4cd1a7"} err="failed to get container status \"4b824fc2837f7daae07bc4855de30aa5f239b6d0a09020d642e3b769dc4cd1a7\": rpc error: code = NotFound desc = could not find container \"4b824fc2837f7daae07bc4855de30aa5f239b6d0a09020d642e3b769dc4cd1a7\": container with ID starting with 4b824fc2837f7daae07bc4855de30aa5f239b6d0a09020d642e3b769dc4cd1a7 not found: ID does not exist"
Sep 11 18:47:57.953458 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.953438    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkn74\" (UniqueName: \"kubernetes.io/projected/e8b8e183-6a12-4a09-b65c-e1ff8299c760-kube-api-access-tkn74\") pod \"e8b8e183-6a12-4a09-b65c-e1ff8299c760\" (UID: \"e8b8e183-6a12-4a09-b65c-e1ff8299c760\") "
Sep 11 18:47:57.953643 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.953514    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b8e183-6a12-4a09-b65c-e1ff8299c760-plugins-config-volume\") pod \"e8b8e183-6a12-4a09-b65c-e1ff8299c760\" (UID: \"e8b8e183-6a12-4a09-b65c-e1ff8299c760\") "
Sep 11 18:47:57.953643 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.953545    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr27w\" (UniqueName: \"kubernetes.io/projected/e5d6cf98-da2d-4b02-ba01-c08cf2955f6f-kube-api-access-cr27w\") pod \"e5d6cf98-da2d-4b02-ba01-c08cf2955f6f\" (UID: \"e5d6cf98-da2d-4b02-ba01-c08cf2955f6f\") "
Sep 11 18:47:57.953643 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.953578    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d6cf98-da2d-4b02-ba01-c08cf2955f6f-plugins-config-volume\") pod \"e5d6cf98-da2d-4b02-ba01-c08cf2955f6f\" (UID: \"e5d6cf98-da2d-4b02-ba01-c08cf2955f6f\") "
Sep 11 18:47:57.953921 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.953900    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b8e183-6a12-4a09-b65c-e1ff8299c760-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8b8e183-6a12-4a09-b65c-e1ff8299c760" (UID: "e8b8e183-6a12-4a09-b65c-e1ff8299c760"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:47:57.954010 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.953928    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d6cf98-da2d-4b02-ba01-c08cf2955f6f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e5d6cf98-da2d-4b02-ba01-c08cf2955f6f" (UID: "e5d6cf98-da2d-4b02-ba01-c08cf2955f6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:47:57.955527 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.955503    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b8e183-6a12-4a09-b65c-e1ff8299c760-kube-api-access-tkn74" (OuterVolumeSpecName: "kube-api-access-tkn74") pod "e8b8e183-6a12-4a09-b65c-e1ff8299c760" (UID: "e8b8e183-6a12-4a09-b65c-e1ff8299c760"). InnerVolumeSpecName "kube-api-access-tkn74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:47:57.955683 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:57.955669    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d6cf98-da2d-4b02-ba01-c08cf2955f6f-kube-api-access-cr27w" (OuterVolumeSpecName: "kube-api-access-cr27w") pod "e5d6cf98-da2d-4b02-ba01-c08cf2955f6f" (UID: "e5d6cf98-da2d-4b02-ba01-c08cf2955f6f"). InnerVolumeSpecName "kube-api-access-cr27w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:47:58.054606 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:58.054532    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5d6cf98-da2d-4b02-ba01-c08cf2955f6f-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:47:58.054606 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:58.054567    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkn74\" (UniqueName: \"kubernetes.io/projected/e8b8e183-6a12-4a09-b65c-e1ff8299c760-kube-api-access-tkn74\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:47:58.054760 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:58.054581    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8b8e183-6a12-4a09-b65c-e1ff8299c760-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:47:58.054760 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:58.054635    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr27w\" (UniqueName: \"kubernetes.io/projected/e5d6cf98-da2d-4b02-ba01-c08cf2955f6f-kube-api-access-cr27w\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:47:58.261315 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:58.261286    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"]
Sep 11 18:47:58.265239 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:58.265205    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-e9c09a-67c5dcd957-zvvhq"]
Sep 11 18:47:58.277751 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:58.277723    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"]
Sep 11 18:47:58.281114 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:58.281094    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-e9c09a-8965c8b6b-xmxk2"]
Sep 11 18:47:59.763934 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:59.763901    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5d6cf98-da2d-4b02-ba01-c08cf2955f6f" path="/var/lib/kubelet/pods/e5d6cf98-da2d-4b02-ba01-c08cf2955f6f/volumes"
Sep 11 18:47:59.764298 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:47:59.764249    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8b8e183-6a12-4a09-b65c-e1ff8299c760" path="/var/lib/kubelet/pods/e8b8e183-6a12-4a09-b65c-e1ff8299c760/volumes"
Sep 11 18:48:02.249157 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:02.249125    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:48:08.369841 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.369804    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5"]
Sep 11 18:48:08.370266 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.370236    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="890d0bc0-d8c6-4814-882c-4116ac0781bc" containerName="revoke-keys"
Sep 11 18:48:08.370266 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.370246    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="890d0bc0-d8c6-4814-882c-4116ac0781bc" containerName="revoke-keys"
Sep 11 18:48:08.370266 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.370265    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8b8e183-6a12-4a09-b65c-e1ff8299c760" containerName="payload-processing"
Sep 11 18:48:08.370408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.370271    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8b8e183-6a12-4a09-b65c-e1ff8299c760" containerName="payload-processing"
Sep 11 18:48:08.370408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.370283    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d6cf98-da2d-4b02-ba01-c08cf2955f6f" containerName="payload-pre-processing"
Sep 11 18:48:08.370408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.370289    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d6cf98-da2d-4b02-ba01-c08cf2955f6f" containerName="payload-pre-processing"
Sep 11 18:48:08.370408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.370360    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d6cf98-da2d-4b02-ba01-c08cf2955f6f" containerName="payload-pre-processing"
Sep 11 18:48:08.370408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.370369    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8b8e183-6a12-4a09-b65c-e1ff8299c760" containerName="payload-processing"
Sep 11 18:48:08.370408 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.370374    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="890d0bc0-d8c6-4814-882c-4116ac0781bc" containerName="revoke-keys"
Sep 11 18:48:08.374705 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.374687    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5"
Sep 11 18:48:08.376827 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.376802    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:48:08.376945 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.376899    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\""
Sep 11 18:48:08.380713 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.380684    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5"]
Sep 11 18:48:08.554430 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.554400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22213e53-a4d2-43b4-b1f4-168bd2937b5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5\" (UID: \"22213e53-a4d2-43b4-b1f4-168bd2937b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5"
Sep 11 18:48:08.655293 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.655266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22213e53-a4d2-43b4-b1f4-168bd2937b5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5\" (UID: \"22213e53-a4d2-43b4-b1f4-168bd2937b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5"
Sep 11 18:48:08.655920 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.655902    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22213e53-a4d2-43b4-b1f4-168bd2937b5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5\" (UID: \"22213e53-a4d2-43b4-b1f4-168bd2937b5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5"
Sep 11 18:48:08.686987 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.686961    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5"
Sep 11 18:48:08.957810 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.957781    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5"]
Sep 11 18:48:08.959967 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:48:08.959942    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22213e53_a4d2_43b4_b1f4_168bd2937b5d.slice/crio-cceee27d850b72b3eda65963ad6f51fa929d502192320adb01e7fc21137f6714 WatchSource:0}: Error finding container cceee27d850b72b3eda65963ad6f51fa929d502192320adb01e7fc21137f6714: Status 404 returned error can't find the container with id cceee27d850b72b3eda65963ad6f51fa929d502192320adb01e7fc21137f6714
Sep 11 18:48:08.996311 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:08.996285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5" event={"ID":"22213e53-a4d2-43b4-b1f4-168bd2937b5d","Type":"ContainerStarted","Data":"cceee27d850b72b3eda65963ad6f51fa929d502192320adb01e7fc21137f6714"}
Sep 11 18:48:10.003005 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:10.002922    2581 generic.go:356] "Generic (PLEG): container finished" podID="22213e53-a4d2-43b4-b1f4-168bd2937b5d" containerID="b2aaacd31f2df2efac699af197607361e9a503573a31046d1b26b15a53417332" exitCode=0
Sep 11 18:48:10.003005 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:10.002972    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5" event={"ID":"22213e53-a4d2-43b4-b1f4-168bd2937b5d","Type":"ContainerDied","Data":"b2aaacd31f2df2efac699af197607361e9a503573a31046d1b26b15a53417332"}
Sep 11 18:48:11.156766 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:11.156741    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5"
Sep 11 18:48:11.176722 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:11.176693    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22213e53-a4d2-43b4-b1f4-168bd2937b5d-maas-api-service-ca\") pod \"22213e53-a4d2-43b4-b1f4-168bd2937b5d\" (UID: \"22213e53-a4d2-43b4-b1f4-168bd2937b5d\") "
Sep 11 18:48:11.177028 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:11.177005    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22213e53-a4d2-43b4-b1f4-168bd2937b5d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22213e53-a4d2-43b4-b1f4-168bd2937b5d" (UID: "22213e53-a4d2-43b4-b1f4-168bd2937b5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:48:11.278239 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:11.278157    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22213e53-a4d2-43b4-b1f4-168bd2937b5d-maas-api-service-ca\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:48:12.015711 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:12.015676    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5" event={"ID":"22213e53-a4d2-43b4-b1f4-168bd2937b5d","Type":"ContainerDied","Data":"cceee27d850b72b3eda65963ad6f51fa929d502192320adb01e7fc21137f6714"}
Sep 11 18:48:12.015711 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:12.015705    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5"
Sep 11 18:48:12.015904 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:12.015725    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cceee27d850b72b3eda65963ad6f51fa929d502192320adb01e7fc21137f6714"
Sep 11 18:48:21.538271 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:21.538188    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:48:23.974123 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:23.974079    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"]
Sep 11 18:48:23.974600 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:23.974343    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk" podUID="7ffb3c02-04d5-4ceb-9988-08f4e9892b89" containerName="payload-processing" containerID="cri-o://c042bcf72017368b1c4a45d2a621fae4226189d573b5a32efa96b07a7d77057b" gracePeriod=30
Sep 11 18:48:23.981438 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:23.981406    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"]
Sep 11 18:48:23.981695 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:23.981649    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64" podUID="fd1c695d-598b-42ad-804d-37d9f01bb909" containerName="payload-pre-processing" containerID="cri-o://0f22d8d3a8d2a7e5898e030ccedb313c31243075935b8c822a0f2cf3b444c984" gracePeriod=30
Sep 11 18:48:24.092532 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.092421    2581 scope.go:117] "RemoveContainer" containerID="aeb060e746fc7d349cddde5660630b9a50f4b9c020285e56611a05bd78ea82de"
Sep 11 18:48:24.103095 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.103067    2581 scope.go:117] "RemoveContainer" containerID="0e92a56342b6e2968e9e92a2e48dbf8b9dc6821f89359c8663d07e5af3a02721"
Sep 11 18:48:24.114741 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.114210    2581 scope.go:117] "RemoveContainer" containerID="1943ed034338fe95406594fcd052586adb2fe8cd83d72b98dddb58331aeccba1"
Sep 11 18:48:24.125738 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.125714    2581 scope.go:117] "RemoveContainer" containerID="d4ce4ced49a87baee373cba9ae6bc34b372bf16124fbcfc6d1fb6a428423c7d0"
Sep 11 18:48:24.161178 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.161139    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5"]
Sep 11 18:48:24.164498 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.164384    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-b0293c-jwnt5"]
Sep 11 18:48:24.241085 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.241030    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"
Sep 11 18:48:24.281554 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.281529    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"
Sep 11 18:48:24.298403 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.298382    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ffb3c02-04d5-4ceb-9988-08f4e9892b89-plugins-config-volume\") pod \"7ffb3c02-04d5-4ceb-9988-08f4e9892b89\" (UID: \"7ffb3c02-04d5-4ceb-9988-08f4e9892b89\") "
Sep 11 18:48:24.298403 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.298420    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nttdp\" (UniqueName: \"kubernetes.io/projected/7ffb3c02-04d5-4ceb-9988-08f4e9892b89-kube-api-access-nttdp\") pod \"7ffb3c02-04d5-4ceb-9988-08f4e9892b89\" (UID: \"7ffb3c02-04d5-4ceb-9988-08f4e9892b89\") "
Sep 11 18:48:24.298946 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.298895    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ffb3c02-04d5-4ceb-9988-08f4e9892b89-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ffb3c02-04d5-4ceb-9988-08f4e9892b89" (UID: "7ffb3c02-04d5-4ceb-9988-08f4e9892b89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:48:24.300963 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.300941    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ffb3c02-04d5-4ceb-9988-08f4e9892b89-kube-api-access-nttdp" (OuterVolumeSpecName: "kube-api-access-nttdp") pod "7ffb3c02-04d5-4ceb-9988-08f4e9892b89" (UID: "7ffb3c02-04d5-4ceb-9988-08f4e9892b89"). InnerVolumeSpecName "kube-api-access-nttdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:48:24.399543 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.399515    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw72b\" (UniqueName: \"kubernetes.io/projected/fd1c695d-598b-42ad-804d-37d9f01bb909-kube-api-access-dw72b\") pod \"fd1c695d-598b-42ad-804d-37d9f01bb909\" (UID: \"fd1c695d-598b-42ad-804d-37d9f01bb909\") "
Sep 11 18:48:24.399676 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.399603    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1c695d-598b-42ad-804d-37d9f01bb909-plugins-config-volume\") pod \"fd1c695d-598b-42ad-804d-37d9f01bb909\" (UID: \"fd1c695d-598b-42ad-804d-37d9f01bb909\") "
Sep 11 18:48:24.399780 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.399768    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ffb3c02-04d5-4ceb-9988-08f4e9892b89-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:48:24.399849 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.399781    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nttdp\" (UniqueName: \"kubernetes.io/projected/7ffb3c02-04d5-4ceb-9988-08f4e9892b89-kube-api-access-nttdp\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:48:24.399978 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.399954    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd1c695d-598b-42ad-804d-37d9f01bb909-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd1c695d-598b-42ad-804d-37d9f01bb909" (UID: "fd1c695d-598b-42ad-804d-37d9f01bb909"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:48:24.401669 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.401648    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd1c695d-598b-42ad-804d-37d9f01bb909-kube-api-access-dw72b" (OuterVolumeSpecName: "kube-api-access-dw72b") pod "fd1c695d-598b-42ad-804d-37d9f01bb909" (UID: "fd1c695d-598b-42ad-804d-37d9f01bb909"). InnerVolumeSpecName "kube-api-access-dw72b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:48:24.500357 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.500284    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd1c695d-598b-42ad-804d-37d9f01bb909-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:48:24.500357 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:24.500318    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw72b\" (UniqueName: \"kubernetes.io/projected/fd1c695d-598b-42ad-804d-37d9f01bb909-kube-api-access-dw72b\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:48:25.087857 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.087823    2581 generic.go:356] "Generic (PLEG): container finished" podID="7ffb3c02-04d5-4ceb-9988-08f4e9892b89" containerID="c042bcf72017368b1c4a45d2a621fae4226189d573b5a32efa96b07a7d77057b" exitCode=0
Sep 11 18:48:25.088268 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.087875    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"
Sep 11 18:48:25.088268 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.087912    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk" event={"ID":"7ffb3c02-04d5-4ceb-9988-08f4e9892b89","Type":"ContainerDied","Data":"c042bcf72017368b1c4a45d2a621fae4226189d573b5a32efa96b07a7d77057b"}
Sep 11 18:48:25.088268 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.087977    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk" event={"ID":"7ffb3c02-04d5-4ceb-9988-08f4e9892b89","Type":"ContainerDied","Data":"687ad6cd6fd78b8e41aa9284837320ed6d2f467c2c85cdffedd63163ffd5105c"}
Sep 11 18:48:25.088268 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.088014    2581 scope.go:117] "RemoveContainer" containerID="c042bcf72017368b1c4a45d2a621fae4226189d573b5a32efa96b07a7d77057b"
Sep 11 18:48:25.089527 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.089270    2581 generic.go:356] "Generic (PLEG): container finished" podID="fd1c695d-598b-42ad-804d-37d9f01bb909" containerID="0f22d8d3a8d2a7e5898e030ccedb313c31243075935b8c822a0f2cf3b444c984" exitCode=0
Sep 11 18:48:25.089527 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.089315    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"
Sep 11 18:48:25.089527 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.089361    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64" event={"ID":"fd1c695d-598b-42ad-804d-37d9f01bb909","Type":"ContainerDied","Data":"0f22d8d3a8d2a7e5898e030ccedb313c31243075935b8c822a0f2cf3b444c984"}
Sep 11 18:48:25.089527 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.089409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64" event={"ID":"fd1c695d-598b-42ad-804d-37d9f01bb909","Type":"ContainerDied","Data":"45b173064fc9646063a7ce26f210498f2cc843036e22719b4a77a785923ee294"}
Sep 11 18:48:25.097180 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.097157    2581 scope.go:117] "RemoveContainer" containerID="c042bcf72017368b1c4a45d2a621fae4226189d573b5a32efa96b07a7d77057b"
Sep 11 18:48:25.097420 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:48:25.097400    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c042bcf72017368b1c4a45d2a621fae4226189d573b5a32efa96b07a7d77057b\": container with ID starting with c042bcf72017368b1c4a45d2a621fae4226189d573b5a32efa96b07a7d77057b not found: ID does not exist" containerID="c042bcf72017368b1c4a45d2a621fae4226189d573b5a32efa96b07a7d77057b"
Sep 11 18:48:25.097495 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.097426    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c042bcf72017368b1c4a45d2a621fae4226189d573b5a32efa96b07a7d77057b"} err="failed to get container status \"c042bcf72017368b1c4a45d2a621fae4226189d573b5a32efa96b07a7d77057b\": rpc error: code = NotFound desc = could not find container \"c042bcf72017368b1c4a45d2a621fae4226189d573b5a32efa96b07a7d77057b\": container with ID starting with c042bcf72017368b1c4a45d2a621fae4226189d573b5a32efa96b07a7d77057b not found: ID does not exist"
Sep 11 18:48:25.097495 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.097443    2581 scope.go:117] "RemoveContainer" containerID="0f22d8d3a8d2a7e5898e030ccedb313c31243075935b8c822a0f2cf3b444c984"
Sep 11 18:48:25.105333 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.105318    2581 scope.go:117] "RemoveContainer" containerID="0f22d8d3a8d2a7e5898e030ccedb313c31243075935b8c822a0f2cf3b444c984"
Sep 11 18:48:25.105602 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:48:25.105587    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f22d8d3a8d2a7e5898e030ccedb313c31243075935b8c822a0f2cf3b444c984\": container with ID starting with 0f22d8d3a8d2a7e5898e030ccedb313c31243075935b8c822a0f2cf3b444c984 not found: ID does not exist" containerID="0f22d8d3a8d2a7e5898e030ccedb313c31243075935b8c822a0f2cf3b444c984"
Sep 11 18:48:25.105646 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.105607    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f22d8d3a8d2a7e5898e030ccedb313c31243075935b8c822a0f2cf3b444c984"} err="failed to get container status \"0f22d8d3a8d2a7e5898e030ccedb313c31243075935b8c822a0f2cf3b444c984\": rpc error: code = NotFound desc = could not find container \"0f22d8d3a8d2a7e5898e030ccedb313c31243075935b8c822a0f2cf3b444c984\": container with ID starting with 0f22d8d3a8d2a7e5898e030ccedb313c31243075935b8c822a0f2cf3b444c984 not found: ID does not exist"
Sep 11 18:48:25.113019 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.112996    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"]
Sep 11 18:48:25.119824 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.119803    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b0293c-578f6c647-dxqjk"]
Sep 11 18:48:25.129688 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.129669    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"]
Sep 11 18:48:25.134770 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.134750    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b0293c-584bd9bd6-pnw64"]
Sep 11 18:48:25.764608 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.764577    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22213e53-a4d2-43b4-b1f4-168bd2937b5d" path="/var/lib/kubelet/pods/22213e53-a4d2-43b4-b1f4-168bd2937b5d/volumes"
Sep 11 18:48:25.765004 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.764987    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ffb3c02-04d5-4ceb-9988-08f4e9892b89" path="/var/lib/kubelet/pods/7ffb3c02-04d5-4ceb-9988-08f4e9892b89/volumes"
Sep 11 18:48:25.765295 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:25.765284    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd1c695d-598b-42ad-804d-37d9f01bb909" path="/var/lib/kubelet/pods/fd1c695d-598b-42ad-804d-37d9f01bb909/volumes"
Sep 11 18:48:29.524002 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:29.523962    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"]
Sep 11 18:48:34.836396 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:34.836363    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:48:44.543939 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.543880    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr" podUID="02d77d70-0b38-4f7e-8011-aa53c61f0b1e" containerName="main" containerID="cri-o://fe166473fe8ce6590417bffd12459f144e116030b16a82b94eb80b04a01c3d46" gracePeriod=45
Sep 11 18:48:44.795890 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.795820    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:48:44.878908 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.878878    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-tmp-dir\") pod \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") "
Sep 11 18:48:44.879068 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.878927    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-tls-certs\") pod \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") "
Sep 11 18:48:44.879068 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.879017    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-dshm\") pod \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") "
Sep 11 18:48:44.879199 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.879072    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-model-cache\") pod \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") "
Sep 11 18:48:44.879199 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.879112    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s99l8\" (UniqueName: \"kubernetes.io/projected/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-kube-api-access-s99l8\") pod \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") "
Sep 11 18:48:44.879199 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.879139    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-home\") pod \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\" (UID: \"02d77d70-0b38-4f7e-8011-aa53c61f0b1e\") "
Sep 11 18:48:44.879199 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.879158    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "02d77d70-0b38-4f7e-8011-aa53c61f0b1e" (UID: "02d77d70-0b38-4f7e-8011-aa53c61f0b1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:48:44.879390 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.879323    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "02d77d70-0b38-4f7e-8011-aa53c61f0b1e" (UID: "02d77d70-0b38-4f7e-8011-aa53c61f0b1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:48:44.879493 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.879438    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-home" (OuterVolumeSpecName: "home") pod "02d77d70-0b38-4f7e-8011-aa53c61f0b1e" (UID: "02d77d70-0b38-4f7e-8011-aa53c61f0b1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:48:44.879615 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.879541    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:48:44.879615 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.879558    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:48:44.879615 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.879570    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:48:44.881272 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.881244    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-dshm" (OuterVolumeSpecName: "dshm") pod "02d77d70-0b38-4f7e-8011-aa53c61f0b1e" (UID: "02d77d70-0b38-4f7e-8011-aa53c61f0b1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:48:44.881370 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.881299    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "02d77d70-0b38-4f7e-8011-aa53c61f0b1e" (UID: "02d77d70-0b38-4f7e-8011-aa53c61f0b1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:48:44.881370 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.881300    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-kube-api-access-s99l8" (OuterVolumeSpecName: "kube-api-access-s99l8") pod "02d77d70-0b38-4f7e-8011-aa53c61f0b1e" (UID: "02d77d70-0b38-4f7e-8011-aa53c61f0b1e"). InnerVolumeSpecName "kube-api-access-s99l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:48:44.980670 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.980646    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:48:44.980670 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.980667    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:48:44.980810 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:44.980676    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s99l8\" (UniqueName: \"kubernetes.io/projected/02d77d70-0b38-4f7e-8011-aa53c61f0b1e-kube-api-access-s99l8\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:48:45.206087 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:45.206044    2581 generic.go:356] "Generic (PLEG): container finished" podID="02d77d70-0b38-4f7e-8011-aa53c61f0b1e" containerID="fe166473fe8ce6590417bffd12459f144e116030b16a82b94eb80b04a01c3d46" exitCode=0
Sep 11 18:48:45.206087 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:45.206086    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"
Sep 11 18:48:45.206448 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:45.206134    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr" event={"ID":"02d77d70-0b38-4f7e-8011-aa53c61f0b1e","Type":"ContainerDied","Data":"fe166473fe8ce6590417bffd12459f144e116030b16a82b94eb80b04a01c3d46"}
Sep 11 18:48:45.206448 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:45.206187    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr" event={"ID":"02d77d70-0b38-4f7e-8011-aa53c61f0b1e","Type":"ContainerDied","Data":"762312c839e9a9c5e7e59898055fb4f86bc76c71b2ecea1a7ce86798048d7833"}
Sep 11 18:48:45.206448 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:45.206204    2581 scope.go:117] "RemoveContainer" containerID="fe166473fe8ce6590417bffd12459f144e116030b16a82b94eb80b04a01c3d46"
Sep 11 18:48:45.216931 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:45.216910    2581 scope.go:117] "RemoveContainer" containerID="fe166473fe8ce6590417bffd12459f144e116030b16a82b94eb80b04a01c3d46"
Sep 11 18:48:45.217212 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:48:45.217191    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe166473fe8ce6590417bffd12459f144e116030b16a82b94eb80b04a01c3d46\": container with ID starting with fe166473fe8ce6590417bffd12459f144e116030b16a82b94eb80b04a01c3d46 not found: ID does not exist" containerID="fe166473fe8ce6590417bffd12459f144e116030b16a82b94eb80b04a01c3d46"
Sep 11 18:48:45.217286 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:45.217218    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe166473fe8ce6590417bffd12459f144e116030b16a82b94eb80b04a01c3d46"} err="failed to get container status \"fe166473fe8ce6590417bffd12459f144e116030b16a82b94eb80b04a01c3d46\": rpc error: code = NotFound desc = could not find container \"fe166473fe8ce6590417bffd12459f144e116030b16a82b94eb80b04a01c3d46\": container with ID starting with fe166473fe8ce6590417bffd12459f144e116030b16a82b94eb80b04a01c3d46 not found: ID does not exist"
Sep 11 18:48:45.228986 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:45.228964    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"]
Sep 11 18:48:45.232946 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:45.232925    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b0293c/ipp-route-b0293c-kserve-54fddf4587-pslkr"]
Sep 11 18:48:45.770328 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:48:45.770296    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02d77d70-0b38-4f7e-8011-aa53c61f0b1e" path="/var/lib/kubelet/pods/02d77d70-0b38-4f7e-8011-aa53c61f0b1e/volumes"
Sep 11 18:49:19.292324 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.292289    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv"]
Sep 11 18:49:19.292819 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.292729    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ffb3c02-04d5-4ceb-9988-08f4e9892b89" containerName="payload-processing"
Sep 11 18:49:19.292819 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.292741    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ffb3c02-04d5-4ceb-9988-08f4e9892b89" containerName="payload-processing"
Sep 11 18:49:19.292819 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.292754    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22213e53-a4d2-43b4-b1f4-168bd2937b5d" containerName="revoke-keys"
Sep 11 18:49:19.292819 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.292760    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="22213e53-a4d2-43b4-b1f4-168bd2937b5d" containerName="revoke-keys"
Sep 11 18:49:19.292819 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.292782    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd1c695d-598b-42ad-804d-37d9f01bb909" containerName="payload-pre-processing"
Sep 11 18:49:19.292819 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.292787    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd1c695d-598b-42ad-804d-37d9f01bb909" containerName="payload-pre-processing"
Sep 11 18:49:19.292819 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.292794    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02d77d70-0b38-4f7e-8011-aa53c61f0b1e" containerName="main"
Sep 11 18:49:19.292819 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.292799    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="02d77d70-0b38-4f7e-8011-aa53c61f0b1e" containerName="main"
Sep 11 18:49:19.293085 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.292856    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd1c695d-598b-42ad-804d-37d9f01bb909" containerName="payload-pre-processing"
Sep 11 18:49:19.293085 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.292865    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="02d77d70-0b38-4f7e-8011-aa53c61f0b1e" containerName="main"
Sep 11 18:49:19.293085 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.292871    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="22213e53-a4d2-43b4-b1f4-168bd2937b5d" containerName="revoke-keys"
Sep 11 18:49:19.293085 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.292877    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ffb3c02-04d5-4ceb-9988-08f4e9892b89" containerName="payload-processing"
Sep 11 18:49:19.296096 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.296080    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv"
Sep 11 18:49:19.297927 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.297907    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:49:19.298031 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.297940    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\""
Sep 11 18:49:19.302418 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.302394    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv"]
Sep 11 18:49:19.389863 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.389837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c33845d-4579-44b3-ae61-0369a739ed63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv\" (UID: \"2c33845d-4579-44b3-ae61-0369a739ed63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv"
Sep 11 18:49:19.491256 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.491229    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c33845d-4579-44b3-ae61-0369a739ed63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv\" (UID: \"2c33845d-4579-44b3-ae61-0369a739ed63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv"
Sep 11 18:49:19.491901 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.491875    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c33845d-4579-44b3-ae61-0369a739ed63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv\" (UID: \"2c33845d-4579-44b3-ae61-0369a739ed63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv"
Sep 11 18:49:19.607953 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.607878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv"
Sep 11 18:49:19.878635 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:19.878609    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv"]
Sep 11 18:49:19.880219 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:49:19.880184    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c33845d_4579_44b3_ae61_0369a739ed63.slice/crio-dacf8d66b58cf48cd86cc0a9e9f541a23a8cc5ce07eb44a801107c96a0b55a62 WatchSource:0}: Error finding container dacf8d66b58cf48cd86cc0a9e9f541a23a8cc5ce07eb44a801107c96a0b55a62: Status 404 returned error can't find the container with id dacf8d66b58cf48cd86cc0a9e9f541a23a8cc5ce07eb44a801107c96a0b55a62
Sep 11 18:49:20.393312 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:20.393274    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv" event={"ID":"2c33845d-4579-44b3-ae61-0369a739ed63","Type":"ContainerStarted","Data":"48624489da40737a771dc362c14403ea43cfb967a49aaafb0ad71ebec38ca2b6"}
Sep 11 18:49:20.393756 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:20.393323    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv" event={"ID":"2c33845d-4579-44b3-ae61-0369a739ed63","Type":"ContainerStarted","Data":"dacf8d66b58cf48cd86cc0a9e9f541a23a8cc5ce07eb44a801107c96a0b55a62"}
Sep 11 18:49:20.409649 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:20.409564    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv" podStartSLOduration=1.4095496779999999 podStartE2EDuration="1.409549678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:20.407626201 +0000 UTC m=+2037.148753043" watchObservedRunningTime="2026-09-11 18:49:20.409549678 +0000 UTC m=+2037.150676521"
Sep 11 18:49:21.399496 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:21.399446    2581 generic.go:356] "Generic (PLEG): container finished" podID="2c33845d-4579-44b3-ae61-0369a739ed63" containerID="48624489da40737a771dc362c14403ea43cfb967a49aaafb0ad71ebec38ca2b6" exitCode=0
Sep 11 18:49:21.399912 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:21.399532    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv" event={"ID":"2c33845d-4579-44b3-ae61-0369a739ed63","Type":"ContainerDied","Data":"48624489da40737a771dc362c14403ea43cfb967a49aaafb0ad71ebec38ca2b6"}
Sep 11 18:49:22.540184 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:22.540157    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv"
Sep 11 18:49:22.621891 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:22.621867    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c33845d-4579-44b3-ae61-0369a739ed63-maas-api-service-ca\") pod \"2c33845d-4579-44b3-ae61-0369a739ed63\" (UID: \"2c33845d-4579-44b3-ae61-0369a739ed63\") "
Sep 11 18:49:22.622200 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:22.622178    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c33845d-4579-44b3-ae61-0369a739ed63-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c33845d-4579-44b3-ae61-0369a739ed63" (UID: "2c33845d-4579-44b3-ae61-0369a739ed63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:49:22.722798 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:22.722730    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c33845d-4579-44b3-ae61-0369a739ed63-maas-api-service-ca\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:49:23.411183 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:23.411155    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv"
Sep 11 18:49:23.411354 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:23.411191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv" event={"ID":"2c33845d-4579-44b3-ae61-0369a739ed63","Type":"ContainerDied","Data":"dacf8d66b58cf48cd86cc0a9e9f541a23a8cc5ce07eb44a801107c96a0b55a62"}
Sep 11 18:49:23.411354 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:23.411231    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dacf8d66b58cf48cd86cc0a9e9f541a23a8cc5ce07eb44a801107c96a0b55a62"
Sep 11 18:49:29.899506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:29.899456    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"]
Sep 11 18:49:29.899999 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:29.899701    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv" podUID="191cb6e8-9601-4a2d-9258-80b84f2ae9ba" containerName="payload-processing" containerID="cri-o://60f094691af6fb70df96e1940ef12cfcbe5955542aeb5ecf3a2ec313afc8a09e" gracePeriod=30
Sep 11 18:49:29.905008 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:29.904979    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"]
Sep 11 18:49:29.905388 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:29.905344    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7" podUID="7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b" containerName="payload-pre-processing" containerID="cri-o://6efb2562cc3c33354b501f41e1cd798c294871e931c114f1837b1a212b9c02a7" gracePeriod=30
Sep 11 18:49:30.083653 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.083619    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv"]
Sep 11 18:49:30.087590 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.087563    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-946072-bcjlv"]
Sep 11 18:49:30.179346 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.179326    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"
Sep 11 18:49:30.182568 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.182548    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"
Sep 11 18:49:30.295287 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.295254    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp8s8\" (UniqueName: \"kubernetes.io/projected/7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b-kube-api-access-tp8s8\") pod \"7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b\" (UID: \"7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b\") "
Sep 11 18:49:30.295456 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.295356    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b-plugins-config-volume\") pod \"7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b\" (UID: \"7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b\") "
Sep 11 18:49:30.295456 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.295395    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/191cb6e8-9601-4a2d-9258-80b84f2ae9ba-plugins-config-volume\") pod \"191cb6e8-9601-4a2d-9258-80b84f2ae9ba\" (UID: \"191cb6e8-9601-4a2d-9258-80b84f2ae9ba\") "
Sep 11 18:49:30.295456 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.295449    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l56pw\" (UniqueName: \"kubernetes.io/projected/191cb6e8-9601-4a2d-9258-80b84f2ae9ba-kube-api-access-l56pw\") pod \"191cb6e8-9601-4a2d-9258-80b84f2ae9ba\" (UID: \"191cb6e8-9601-4a2d-9258-80b84f2ae9ba\") "
Sep 11 18:49:30.295761 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.295739    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b" (UID: "7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:49:30.295828 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.295812    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191cb6e8-9601-4a2d-9258-80b84f2ae9ba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "191cb6e8-9601-4a2d-9258-80b84f2ae9ba" (UID: "191cb6e8-9601-4a2d-9258-80b84f2ae9ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:49:30.297596 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.297566    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b-kube-api-access-tp8s8" (OuterVolumeSpecName: "kube-api-access-tp8s8") pod "7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b" (UID: "7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b"). InnerVolumeSpecName "kube-api-access-tp8s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:49:30.297698 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.297685    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191cb6e8-9601-4a2d-9258-80b84f2ae9ba-kube-api-access-l56pw" (OuterVolumeSpecName: "kube-api-access-l56pw") pod "191cb6e8-9601-4a2d-9258-80b84f2ae9ba" (UID: "191cb6e8-9601-4a2d-9258-80b84f2ae9ba"). InnerVolumeSpecName "kube-api-access-l56pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:49:30.396840 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.396812    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp8s8\" (UniqueName: \"kubernetes.io/projected/7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b-kube-api-access-tp8s8\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:49:30.396840 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.396835    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:49:30.396840 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.396844    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/191cb6e8-9601-4a2d-9258-80b84f2ae9ba-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:49:30.397038 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.396853    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l56pw\" (UniqueName: \"kubernetes.io/projected/191cb6e8-9601-4a2d-9258-80b84f2ae9ba-kube-api-access-l56pw\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:49:30.449806 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.449735    2581 generic.go:356] "Generic (PLEG): container finished" podID="7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b" containerID="6efb2562cc3c33354b501f41e1cd798c294871e931c114f1837b1a212b9c02a7" exitCode=0
Sep 11 18:49:30.449806 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.449802    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"
Sep 11 18:49:30.449982 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.449826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7" event={"ID":"7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b","Type":"ContainerDied","Data":"6efb2562cc3c33354b501f41e1cd798c294871e931c114f1837b1a212b9c02a7"}
Sep 11 18:49:30.449982 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.449910    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7" event={"ID":"7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b","Type":"ContainerDied","Data":"1262ea5f21d92df0db68b840ce43c60b0c477d4e8c774f6b1c243475ef951eda"}
Sep 11 18:49:30.449982 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.449936    2581 scope.go:117] "RemoveContainer" containerID="6efb2562cc3c33354b501f41e1cd798c294871e931c114f1837b1a212b9c02a7"
Sep 11 18:49:30.451153 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.451130    2581 generic.go:356] "Generic (PLEG): container finished" podID="191cb6e8-9601-4a2d-9258-80b84f2ae9ba" containerID="60f094691af6fb70df96e1940ef12cfcbe5955542aeb5ecf3a2ec313afc8a09e" exitCode=0
Sep 11 18:49:30.451237 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.451173    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv" event={"ID":"191cb6e8-9601-4a2d-9258-80b84f2ae9ba","Type":"ContainerDied","Data":"60f094691af6fb70df96e1940ef12cfcbe5955542aeb5ecf3a2ec313afc8a09e"}
Sep 11 18:49:30.451237 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.451221    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv" event={"ID":"191cb6e8-9601-4a2d-9258-80b84f2ae9ba","Type":"ContainerDied","Data":"155d795778f3b181d8d2910296db376cb2934a8470710b48763492043a55b195"}
Sep 11 18:49:30.451315 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.451184    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"
Sep 11 18:49:30.460403 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.460385    2581 scope.go:117] "RemoveContainer" containerID="6efb2562cc3c33354b501f41e1cd798c294871e931c114f1837b1a212b9c02a7"
Sep 11 18:49:30.460744 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:49:30.460720    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6efb2562cc3c33354b501f41e1cd798c294871e931c114f1837b1a212b9c02a7\": container with ID starting with 6efb2562cc3c33354b501f41e1cd798c294871e931c114f1837b1a212b9c02a7 not found: ID does not exist" containerID="6efb2562cc3c33354b501f41e1cd798c294871e931c114f1837b1a212b9c02a7"
Sep 11 18:49:30.460851 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.460751    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6efb2562cc3c33354b501f41e1cd798c294871e931c114f1837b1a212b9c02a7"} err="failed to get container status \"6efb2562cc3c33354b501f41e1cd798c294871e931c114f1837b1a212b9c02a7\": rpc error: code = NotFound desc = could not find container \"6efb2562cc3c33354b501f41e1cd798c294871e931c114f1837b1a212b9c02a7\": container with ID starting with 6efb2562cc3c33354b501f41e1cd798c294871e931c114f1837b1a212b9c02a7 not found: ID does not exist"
Sep 11 18:49:30.460851 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.460770    2581 scope.go:117] "RemoveContainer" containerID="60f094691af6fb70df96e1940ef12cfcbe5955542aeb5ecf3a2ec313afc8a09e"
Sep 11 18:49:30.468765 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.468750    2581 scope.go:117] "RemoveContainer" containerID="60f094691af6fb70df96e1940ef12cfcbe5955542aeb5ecf3a2ec313afc8a09e"
Sep 11 18:49:30.469006 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:49:30.468989    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f094691af6fb70df96e1940ef12cfcbe5955542aeb5ecf3a2ec313afc8a09e\": container with ID starting with 60f094691af6fb70df96e1940ef12cfcbe5955542aeb5ecf3a2ec313afc8a09e not found: ID does not exist" containerID="60f094691af6fb70df96e1940ef12cfcbe5955542aeb5ecf3a2ec313afc8a09e"
Sep 11 18:49:30.469055 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.469013    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f094691af6fb70df96e1940ef12cfcbe5955542aeb5ecf3a2ec313afc8a09e"} err="failed to get container status \"60f094691af6fb70df96e1940ef12cfcbe5955542aeb5ecf3a2ec313afc8a09e\": rpc error: code = NotFound desc = could not find container \"60f094691af6fb70df96e1940ef12cfcbe5955542aeb5ecf3a2ec313afc8a09e\": container with ID starting with 60f094691af6fb70df96e1940ef12cfcbe5955542aeb5ecf3a2ec313afc8a09e not found: ID does not exist"
Sep 11 18:49:30.475328 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.475307    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"]
Sep 11 18:49:30.481334 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.481316    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-946072-6b446d8487-prtdv"]
Sep 11 18:49:30.491029 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.491005    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"]
Sep 11 18:49:30.495695 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:30.495674    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-946072-6995f4c774-jt9j7"]
Sep 11 18:49:31.765017 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:31.764984    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="191cb6e8-9601-4a2d-9258-80b84f2ae9ba" path="/var/lib/kubelet/pods/191cb6e8-9601-4a2d-9258-80b84f2ae9ba/volumes"
Sep 11 18:49:31.765402 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:31.765301    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c33845d-4579-44b3-ae61-0369a739ed63" path="/var/lib/kubelet/pods/2c33845d-4579-44b3-ae61-0369a739ed63/volumes"
Sep 11 18:49:31.765676 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:31.765661    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b" path="/var/lib/kubelet/pods/7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b/volumes"
Sep 11 18:49:33.451794 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:33.451753    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:49:37.270309 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.270274    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85"]
Sep 11 18:49:37.270780 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.270669    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c33845d-4579-44b3-ae61-0369a739ed63" containerName="revoke-keys"
Sep 11 18:49:37.270780 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.270680    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c33845d-4579-44b3-ae61-0369a739ed63" containerName="revoke-keys"
Sep 11 18:49:37.270780 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.270701    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="191cb6e8-9601-4a2d-9258-80b84f2ae9ba" containerName="payload-processing"
Sep 11 18:49:37.270780 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.270707    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="191cb6e8-9601-4a2d-9258-80b84f2ae9ba" containerName="payload-processing"
Sep 11 18:49:37.270780 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.270721    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b" containerName="payload-pre-processing"
Sep 11 18:49:37.270780 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.270726    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b" containerName="payload-pre-processing"
Sep 11 18:49:37.270780 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.270782    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="191cb6e8-9601-4a2d-9258-80b84f2ae9ba" containerName="payload-processing"
Sep 11 18:49:37.271000 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.270792    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e2384c0-db5e-4e9a-87d3-4bfa02cdde5b" containerName="payload-pre-processing"
Sep 11 18:49:37.271000 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.270799    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c33845d-4579-44b3-ae61-0369a739ed63" containerName="revoke-keys"
Sep 11 18:49:37.275404 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.275384    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85"
Sep 11 18:49:37.277362 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.277341    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:49:37.277471 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.277458    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\""
Sep 11 18:49:37.281219 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.281193    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85"]
Sep 11 18:49:37.356015 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.355988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72211a7b-20ff-4681-b545-35d6ba8f9c7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85\" (UID: \"72211a7b-20ff-4681-b545-35d6ba8f9c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85"
Sep 11 18:49:37.456807 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.456775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72211a7b-20ff-4681-b545-35d6ba8f9c7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85\" (UID: \"72211a7b-20ff-4681-b545-35d6ba8f9c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85"
Sep 11 18:49:37.457370 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.457349    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72211a7b-20ff-4681-b545-35d6ba8f9c7d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85\" (UID: \"72211a7b-20ff-4681-b545-35d6ba8f9c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85"
Sep 11 18:49:37.588383 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.588302    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85"
Sep 11 18:49:37.858681 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:37.858653    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85"]
Sep 11 18:49:37.860241 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:49:37.860217    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72211a7b_20ff_4681_b545_35d6ba8f9c7d.slice/crio-7352645a95bc3dd41425b1f0444737bad9b7f3d638ccc096f9bf1389cfaff238 WatchSource:0}: Error finding container 7352645a95bc3dd41425b1f0444737bad9b7f3d638ccc096f9bf1389cfaff238: Status 404 returned error can't find the container with id 7352645a95bc3dd41425b1f0444737bad9b7f3d638ccc096f9bf1389cfaff238
Sep 11 18:49:38.499410 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:38.499371    2581 generic.go:356] "Generic (PLEG): container finished" podID="72211a7b-20ff-4681-b545-35d6ba8f9c7d" containerID="aa90ef568d285463a9c463579edb8d9f90d2580c85841167d9e5164ec2f7e7c7" exitCode=0
Sep 11 18:49:38.499749 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:38.499453    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85" event={"ID":"72211a7b-20ff-4681-b545-35d6ba8f9c7d","Type":"ContainerDied","Data":"aa90ef568d285463a9c463579edb8d9f90d2580c85841167d9e5164ec2f7e7c7"}
Sep 11 18:49:38.499749 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:38.499534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85" event={"ID":"72211a7b-20ff-4681-b545-35d6ba8f9c7d","Type":"ContainerStarted","Data":"7352645a95bc3dd41425b1f0444737bad9b7f3d638ccc096f9bf1389cfaff238"}
Sep 11 18:49:39.635745 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:39.635712    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85"
Sep 11 18:49:39.673749 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:39.673722    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72211a7b-20ff-4681-b545-35d6ba8f9c7d-maas-api-service-ca\") pod \"72211a7b-20ff-4681-b545-35d6ba8f9c7d\" (UID: \"72211a7b-20ff-4681-b545-35d6ba8f9c7d\") "
Sep 11 18:49:39.674076 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:39.674053    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72211a7b-20ff-4681-b545-35d6ba8f9c7d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72211a7b-20ff-4681-b545-35d6ba8f9c7d" (UID: "72211a7b-20ff-4681-b545-35d6ba8f9c7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:49:39.774653 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:39.774628    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72211a7b-20ff-4681-b545-35d6ba8f9c7d-maas-api-service-ca\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:49:40.511893 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:40.511860    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85"
Sep 11 18:49:40.512066 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:40.511895    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85" event={"ID":"72211a7b-20ff-4681-b545-35d6ba8f9c7d","Type":"ContainerDied","Data":"7352645a95bc3dd41425b1f0444737bad9b7f3d638ccc096f9bf1389cfaff238"}
Sep 11 18:49:40.512066 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:40.511960    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7352645a95bc3dd41425b1f0444737bad9b7f3d638ccc096f9bf1389cfaff238"
Sep 11 18:49:47.894060 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:47.893983    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7ca6df-29819205-fr52q"]
Sep 11 18:49:47.896645 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:47.896620    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-7ca6df-29819205-fr52q"]
Sep 11 18:49:47.933298 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:47.933270    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"]
Sep 11 18:49:47.933542 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:47.933518    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6" podUID="6e45883f-542d-4cdb-83bb-bf5a063ff3d1" containerName="payload-pre-processing" containerID="cri-o://7af88e4495da4252afad9345e162187fa98204e1d735162be184f5d089e61a84" gracePeriod=30
Sep 11 18:49:48.125706 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.125677    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85"]
Sep 11 18:49:48.128516 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.128458    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-7ca6df-j2h85"]
Sep 11 18:49:48.177995 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.177976    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"
Sep 11 18:49:48.250934 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.250907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e45883f-542d-4cdb-83bb-bf5a063ff3d1-plugins-config-volume\") pod \"6e45883f-542d-4cdb-83bb-bf5a063ff3d1\" (UID: \"6e45883f-542d-4cdb-83bb-bf5a063ff3d1\") "
Sep 11 18:49:48.251091 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.250959    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svsst\" (UniqueName: \"kubernetes.io/projected/6e45883f-542d-4cdb-83bb-bf5a063ff3d1-kube-api-access-svsst\") pod \"6e45883f-542d-4cdb-83bb-bf5a063ff3d1\" (UID: \"6e45883f-542d-4cdb-83bb-bf5a063ff3d1\") "
Sep 11 18:49:48.251294 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.251267    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e45883f-542d-4cdb-83bb-bf5a063ff3d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e45883f-542d-4cdb-83bb-bf5a063ff3d1" (UID: "6e45883f-542d-4cdb-83bb-bf5a063ff3d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:49:48.253180 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.253156    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e45883f-542d-4cdb-83bb-bf5a063ff3d1-kube-api-access-svsst" (OuterVolumeSpecName: "kube-api-access-svsst") pod "6e45883f-542d-4cdb-83bb-bf5a063ff3d1" (UID: "6e45883f-542d-4cdb-83bb-bf5a063ff3d1"). InnerVolumeSpecName "kube-api-access-svsst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:49:48.352593 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.352569    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e45883f-542d-4cdb-83bb-bf5a063ff3d1-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:49:48.352593 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.352591    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svsst\" (UniqueName: \"kubernetes.io/projected/6e45883f-542d-4cdb-83bb-bf5a063ff3d1-kube-api-access-svsst\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:49:48.554640 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.554556    2581 generic.go:356] "Generic (PLEG): container finished" podID="6e45883f-542d-4cdb-83bb-bf5a063ff3d1" containerID="7af88e4495da4252afad9345e162187fa98204e1d735162be184f5d089e61a84" exitCode=0
Sep 11 18:49:48.554640 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.554604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6" event={"ID":"6e45883f-542d-4cdb-83bb-bf5a063ff3d1","Type":"ContainerDied","Data":"7af88e4495da4252afad9345e162187fa98204e1d735162be184f5d089e61a84"}
Sep 11 18:49:48.554822 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.554645    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6" event={"ID":"6e45883f-542d-4cdb-83bb-bf5a063ff3d1","Type":"ContainerDied","Data":"8440bfcddc1b818d1b549b32bb41e6b25b05d9be93b07a1fe7612d20e053d7dc"}
Sep 11 18:49:48.554822 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.554665    2581 scope.go:117] "RemoveContainer" containerID="7af88e4495da4252afad9345e162187fa98204e1d735162be184f5d089e61a84"
Sep 11 18:49:48.554822 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.554607    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"
Sep 11 18:49:48.564585 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.564567    2581 scope.go:117] "RemoveContainer" containerID="7af88e4495da4252afad9345e162187fa98204e1d735162be184f5d089e61a84"
Sep 11 18:49:48.564837 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:49:48.564818    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af88e4495da4252afad9345e162187fa98204e1d735162be184f5d089e61a84\": container with ID starting with 7af88e4495da4252afad9345e162187fa98204e1d735162be184f5d089e61a84 not found: ID does not exist" containerID="7af88e4495da4252afad9345e162187fa98204e1d735162be184f5d089e61a84"
Sep 11 18:49:48.564891 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.564846    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af88e4495da4252afad9345e162187fa98204e1d735162be184f5d089e61a84"} err="failed to get container status \"7af88e4495da4252afad9345e162187fa98204e1d735162be184f5d089e61a84\": rpc error: code = NotFound desc = could not find container \"7af88e4495da4252afad9345e162187fa98204e1d735162be184f5d089e61a84\": container with ID starting with 7af88e4495da4252afad9345e162187fa98204e1d735162be184f5d089e61a84 not found: ID does not exist"
Sep 11 18:49:48.577947 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.577925    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"]
Sep 11 18:49:48.589315 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:48.589290    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-7ca6df-8565b4c6b6-gndz6"]
Sep 11 18:49:49.764086 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:49.764051    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e45883f-542d-4cdb-83bb-bf5a063ff3d1" path="/var/lib/kubelet/pods/6e45883f-542d-4cdb-83bb-bf5a063ff3d1/volumes"
Sep 11 18:49:49.764523 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:49.764352    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72211a7b-20ff-4681-b545-35d6ba8f9c7d" path="/var/lib/kubelet/pods/72211a7b-20ff-4681-b545-35d6ba8f9c7d/volumes"
Sep 11 18:49:49.764782 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:49.764768    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ddbddb1-f3bf-4fb3-b20f-d4360d745c55" path="/var/lib/kubelet/pods/7ddbddb1-f3bf-4fb3-b20f-d4360d745c55/volumes"
Sep 11 18:49:51.038392 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:51.038356    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:49:55.623538 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:55.623505    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk"]
Sep 11 18:49:55.623910 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:55.623867    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e45883f-542d-4cdb-83bb-bf5a063ff3d1" containerName="payload-pre-processing"
Sep 11 18:49:55.623910 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:55.623878    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e45883f-542d-4cdb-83bb-bf5a063ff3d1" containerName="payload-pre-processing"
Sep 11 18:49:55.623910 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:55.623895    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72211a7b-20ff-4681-b545-35d6ba8f9c7d" containerName="revoke-keys"
Sep 11 18:49:55.623910 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:55.623901    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="72211a7b-20ff-4681-b545-35d6ba8f9c7d" containerName="revoke-keys"
Sep 11 18:49:55.624044 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:55.623963    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e45883f-542d-4cdb-83bb-bf5a063ff3d1" containerName="payload-pre-processing"
Sep 11 18:49:55.624044 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:55.623976    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="72211a7b-20ff-4681-b545-35d6ba8f9c7d" containerName="revoke-keys"
Sep 11 18:49:55.628457 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:55.628442    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk"
Sep 11 18:49:55.630401 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:55.630369    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n968b\""
Sep 11 18:49:55.630562 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:55.630450    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:49:55.640738 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:55.640718    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk"]
Sep 11 18:49:55.711053 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:55.711026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25edfbe6-858a-4c6c-8507-3cc901e4ad3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk\" (UID: \"25edfbe6-858a-4c6c-8507-3cc901e4ad3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk"
Sep 11 18:49:55.812386 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:55.812355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25edfbe6-858a-4c6c-8507-3cc901e4ad3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk\" (UID: \"25edfbe6-858a-4c6c-8507-3cc901e4ad3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk"
Sep 11 18:49:55.812950 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:55.812932    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25edfbe6-858a-4c6c-8507-3cc901e4ad3f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk\" (UID: \"25edfbe6-858a-4c6c-8507-3cc901e4ad3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk"
Sep 11 18:49:55.938067 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:55.938039    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk"
Sep 11 18:49:56.211409 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:56.211334    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk"]
Sep 11 18:49:56.596532 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:56.596504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk" event={"ID":"25edfbe6-858a-4c6c-8507-3cc901e4ad3f","Type":"ContainerStarted","Data":"f0dc1b94d14529ff98f3d94f9273d25d0ea702f1c7002969e832d9c95645f486"}
Sep 11 18:49:56.596628 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:56.596555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk" event={"ID":"25edfbe6-858a-4c6c-8507-3cc901e4ad3f","Type":"ContainerStarted","Data":"62b7b40db95dbc965bcdc37ca0eb8e41e2a9e516ef3f5cab8926a8174e942465"}
Sep 11 18:49:56.616673 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:56.616632    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk" podStartSLOduration=1.6166161049999999 podStartE2EDuration="1.616616105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:56.615308694 +0000 UTC m=+2073.356435539" watchObservedRunningTime="2026-09-11 18:49:56.616616105 +0000 UTC m=+2073.357742988"
Sep 11 18:49:57.603201 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:57.603171    2581 generic.go:356] "Generic (PLEG): container finished" podID="25edfbe6-858a-4c6c-8507-3cc901e4ad3f" containerID="f0dc1b94d14529ff98f3d94f9273d25d0ea702f1c7002969e832d9c95645f486" exitCode=0
Sep 11 18:49:57.603606 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:57.603254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk" event={"ID":"25edfbe6-858a-4c6c-8507-3cc901e4ad3f","Type":"ContainerDied","Data":"f0dc1b94d14529ff98f3d94f9273d25d0ea702f1c7002969e832d9c95645f486"}
Sep 11 18:49:58.741220 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:58.741197    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk"
Sep 11 18:49:58.840145 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:58.840115    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25edfbe6-858a-4c6c-8507-3cc901e4ad3f-maas-api-service-ca\") pod \"25edfbe6-858a-4c6c-8507-3cc901e4ad3f\" (UID: \"25edfbe6-858a-4c6c-8507-3cc901e4ad3f\") "
Sep 11 18:49:58.840448 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:58.840428    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25edfbe6-858a-4c6c-8507-3cc901e4ad3f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "25edfbe6-858a-4c6c-8507-3cc901e4ad3f" (UID: "25edfbe6-858a-4c6c-8507-3cc901e4ad3f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:49:58.941558 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:58.941536    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25edfbe6-858a-4c6c-8507-3cc901e4ad3f-maas-api-service-ca\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:49:59.615592 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:59.615561    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk"
Sep 11 18:49:59.615772 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:59.615608    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk" event={"ID":"25edfbe6-858a-4c6c-8507-3cc901e4ad3f","Type":"ContainerDied","Data":"62b7b40db95dbc965bcdc37ca0eb8e41e2a9e516ef3f5cab8926a8174e942465"}
Sep 11 18:49:59.615772 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:49:59.615648    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62b7b40db95dbc965bcdc37ca0eb8e41e2a9e516ef3f5cab8926a8174e942465"
Sep 11 18:50:06.184296 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.184256    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"]
Sep 11 18:50:06.184747 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.184532    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl" podUID="2e455243-6075-4137-ba60-a5b81dae2c3e" containerName="payload-processing" containerID="cri-o://180884c3d311e815d7ed5c2e59c233f5c287fec36c8af06befd7450122b2fd40" gracePeriod=30
Sep 11 18:50:06.193645 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.193530    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"]
Sep 11 18:50:06.193814 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.193782    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv" podUID="0fea0428-cb51-4a39-b775-e0f8833b3e1b" containerName="payload-pre-processing" containerID="cri-o://740fb34f4a21e17c5752e329c929fc516a51ec197a22299849679c314bef8011" gracePeriod=30
Sep 11 18:50:06.378303 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.378269    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk"]
Sep 11 18:50:06.381863 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.381833    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-711a34-jl7tk"]
Sep 11 18:50:06.445912 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.445849    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"
Sep 11 18:50:06.477168 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.477141    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"
Sep 11 18:50:06.505726 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.505700    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bfwz\" (UniqueName: \"kubernetes.io/projected/2e455243-6075-4137-ba60-a5b81dae2c3e-kube-api-access-5bfwz\") pod \"2e455243-6075-4137-ba60-a5b81dae2c3e\" (UID: \"2e455243-6075-4137-ba60-a5b81dae2c3e\") "
Sep 11 18:50:06.505874 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.505759    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e455243-6075-4137-ba60-a5b81dae2c3e-plugins-config-volume\") pod \"2e455243-6075-4137-ba60-a5b81dae2c3e\" (UID: \"2e455243-6075-4137-ba60-a5b81dae2c3e\") "
Sep 11 18:50:06.506221 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.506191    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e455243-6075-4137-ba60-a5b81dae2c3e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2e455243-6075-4137-ba60-a5b81dae2c3e" (UID: "2e455243-6075-4137-ba60-a5b81dae2c3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:50:06.507877 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.507853    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e455243-6075-4137-ba60-a5b81dae2c3e-kube-api-access-5bfwz" (OuterVolumeSpecName: "kube-api-access-5bfwz") pod "2e455243-6075-4137-ba60-a5b81dae2c3e" (UID: "2e455243-6075-4137-ba60-a5b81dae2c3e"). InnerVolumeSpecName "kube-api-access-5bfwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:06.606728 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.606697    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wngt4\" (UniqueName: \"kubernetes.io/projected/0fea0428-cb51-4a39-b775-e0f8833b3e1b-kube-api-access-wngt4\") pod \"0fea0428-cb51-4a39-b775-e0f8833b3e1b\" (UID: \"0fea0428-cb51-4a39-b775-e0f8833b3e1b\") "
Sep 11 18:50:06.606909 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.606821    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fea0428-cb51-4a39-b775-e0f8833b3e1b-plugins-config-volume\") pod \"0fea0428-cb51-4a39-b775-e0f8833b3e1b\" (UID: \"0fea0428-cb51-4a39-b775-e0f8833b3e1b\") "
Sep 11 18:50:06.607106 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.607089    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e455243-6075-4137-ba60-a5b81dae2c3e-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:50:06.607162 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.607109    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bfwz\" (UniqueName: \"kubernetes.io/projected/2e455243-6075-4137-ba60-a5b81dae2c3e-kube-api-access-5bfwz\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:50:06.607162 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.607131    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fea0428-cb51-4a39-b775-e0f8833b3e1b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0fea0428-cb51-4a39-b775-e0f8833b3e1b" (UID: "0fea0428-cb51-4a39-b775-e0f8833b3e1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:50:06.608701 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.608681    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fea0428-cb51-4a39-b775-e0f8833b3e1b-kube-api-access-wngt4" (OuterVolumeSpecName: "kube-api-access-wngt4") pod "0fea0428-cb51-4a39-b775-e0f8833b3e1b" (UID: "0fea0428-cb51-4a39-b775-e0f8833b3e1b"). InnerVolumeSpecName "kube-api-access-wngt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:06.651066 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.651030    2581 generic.go:356] "Generic (PLEG): container finished" podID="0fea0428-cb51-4a39-b775-e0f8833b3e1b" containerID="740fb34f4a21e17c5752e329c929fc516a51ec197a22299849679c314bef8011" exitCode=0
Sep 11 18:50:06.651190 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.651081    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"
Sep 11 18:50:06.651190 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.651122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv" event={"ID":"0fea0428-cb51-4a39-b775-e0f8833b3e1b","Type":"ContainerDied","Data":"740fb34f4a21e17c5752e329c929fc516a51ec197a22299849679c314bef8011"}
Sep 11 18:50:06.651293 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.651201    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv" event={"ID":"0fea0428-cb51-4a39-b775-e0f8833b3e1b","Type":"ContainerDied","Data":"f0e83e8bdb12a1b02116826a6647a76a5ece92db271206bea2136f6fc6c7b47b"}
Sep 11 18:50:06.651293 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.651228    2581 scope.go:117] "RemoveContainer" containerID="740fb34f4a21e17c5752e329c929fc516a51ec197a22299849679c314bef8011"
Sep 11 18:50:06.652705 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.652673    2581 generic.go:356] "Generic (PLEG): container finished" podID="2e455243-6075-4137-ba60-a5b81dae2c3e" containerID="180884c3d311e815d7ed5c2e59c233f5c287fec36c8af06befd7450122b2fd40" exitCode=0
Sep 11 18:50:06.652805 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.652717    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"
Sep 11 18:50:06.652805 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.652725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl" event={"ID":"2e455243-6075-4137-ba60-a5b81dae2c3e","Type":"ContainerDied","Data":"180884c3d311e815d7ed5c2e59c233f5c287fec36c8af06befd7450122b2fd40"}
Sep 11 18:50:06.652805 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.652757    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl" event={"ID":"2e455243-6075-4137-ba60-a5b81dae2c3e","Type":"ContainerDied","Data":"3a429a6c06e2d326ece2af1db8950611a0168e651a1852fa3e44b005df724382"}
Sep 11 18:50:06.661124 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.661108    2581 scope.go:117] "RemoveContainer" containerID="740fb34f4a21e17c5752e329c929fc516a51ec197a22299849679c314bef8011"
Sep 11 18:50:06.661375 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:50:06.661358    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"740fb34f4a21e17c5752e329c929fc516a51ec197a22299849679c314bef8011\": container with ID starting with 740fb34f4a21e17c5752e329c929fc516a51ec197a22299849679c314bef8011 not found: ID does not exist" containerID="740fb34f4a21e17c5752e329c929fc516a51ec197a22299849679c314bef8011"
Sep 11 18:50:06.661447 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.661381    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"740fb34f4a21e17c5752e329c929fc516a51ec197a22299849679c314bef8011"} err="failed to get container status \"740fb34f4a21e17c5752e329c929fc516a51ec197a22299849679c314bef8011\": rpc error: code = NotFound desc = could not find container \"740fb34f4a21e17c5752e329c929fc516a51ec197a22299849679c314bef8011\": container with ID starting with 740fb34f4a21e17c5752e329c929fc516a51ec197a22299849679c314bef8011 not found: ID does not exist"
Sep 11 18:50:06.661447 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.661397    2581 scope.go:117] "RemoveContainer" containerID="180884c3d311e815d7ed5c2e59c233f5c287fec36c8af06befd7450122b2fd40"
Sep 11 18:50:06.670333 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.670317    2581 scope.go:117] "RemoveContainer" containerID="180884c3d311e815d7ed5c2e59c233f5c287fec36c8af06befd7450122b2fd40"
Sep 11 18:50:06.670655 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:50:06.670634    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"180884c3d311e815d7ed5c2e59c233f5c287fec36c8af06befd7450122b2fd40\": container with ID starting with 180884c3d311e815d7ed5c2e59c233f5c287fec36c8af06befd7450122b2fd40 not found: ID does not exist" containerID="180884c3d311e815d7ed5c2e59c233f5c287fec36c8af06befd7450122b2fd40"
Sep 11 18:50:06.670727 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.670663    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180884c3d311e815d7ed5c2e59c233f5c287fec36c8af06befd7450122b2fd40"} err="failed to get container status \"180884c3d311e815d7ed5c2e59c233f5c287fec36c8af06befd7450122b2fd40\": rpc error: code = NotFound desc = could not find container \"180884c3d311e815d7ed5c2e59c233f5c287fec36c8af06befd7450122b2fd40\": container with ID starting with 180884c3d311e815d7ed5c2e59c233f5c287fec36c8af06befd7450122b2fd40 not found: ID does not exist"
Sep 11 18:50:06.676974 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.676953    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"]
Sep 11 18:50:06.681686 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.681666    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-711a34-694ff46dcf-b4vgl"]
Sep 11 18:50:06.692491 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.692452    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"]
Sep 11 18:50:06.695113 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.695094    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-711a34-56d7c574db-b4ljv"]
Sep 11 18:50:06.707906 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.707857    2581 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fea0428-cb51-4a39-b775-e0f8833b3e1b-plugins-config-volume\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:50:06.707906 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:06.707878    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wngt4\" (UniqueName: \"kubernetes.io/projected/0fea0428-cb51-4a39-b775-e0f8833b3e1b-kube-api-access-wngt4\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:50:07.763897 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:07.763864    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fea0428-cb51-4a39-b775-e0f8833b3e1b" path="/var/lib/kubelet/pods/0fea0428-cb51-4a39-b775-e0f8833b3e1b/volumes"
Sep 11 18:50:07.764267 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:07.764220    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25edfbe6-858a-4c6c-8507-3cc901e4ad3f" path="/var/lib/kubelet/pods/25edfbe6-858a-4c6c-8507-3cc901e4ad3f/volumes"
Sep 11 18:50:07.764593 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:07.764578    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e455243-6075-4137-ba60-a5b81dae2c3e" path="/var/lib/kubelet/pods/2e455243-6075-4137-ba60-a5b81dae2c3e/volumes"
Sep 11 18:50:08.739317 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:08.739285    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:50:17.942638 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:50:17.942603    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:51:24.225448 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:51:24.225356    2581 scope.go:117] "RemoveContainer" containerID="140f2790ad31ff3043105bd746d000588ef75de5e4c2f75c6c812d34b7d8fa12"
Sep 11 18:51:39.946048 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:51:39.946003    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:52:05.863539 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:52:05.863498    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:52:13.843911 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:52:13.843874    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:52:23.040662 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:52:23.040622    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:52:30.344071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:52:30.344034    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:52:41.750202 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:52:41.750128    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:52:47.945304 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:52:47.945204    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:54:14.238014 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:14.237976    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:54:22.443089 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:22.443001    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:54:24.261821 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:24.261782    2581 scope.go:117] "RemoveContainer" containerID="b2aaacd31f2df2efac699af197607361e9a503573a31046d1b26b15a53417332"
Sep 11 18:54:24.270211 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:24.270190    2581 scope.go:117] "RemoveContainer" containerID="da90a63bd7974bf61013e6ffdfb585a5d5a7f26e944ecb43f6d50d900d2df270"
Sep 11 18:54:26.343164 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.343118    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"]
Sep 11 18:54:26.343663 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.343504    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e455243-6075-4137-ba60-a5b81dae2c3e" containerName="payload-processing"
Sep 11 18:54:26.343663 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.343519    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e455243-6075-4137-ba60-a5b81dae2c3e" containerName="payload-processing"
Sep 11 18:54:26.343663 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.343525    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fea0428-cb51-4a39-b775-e0f8833b3e1b" containerName="payload-pre-processing"
Sep 11 18:54:26.343663 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.343531    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fea0428-cb51-4a39-b775-e0f8833b3e1b" containerName="payload-pre-processing"
Sep 11 18:54:26.343663 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.343547    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25edfbe6-858a-4c6c-8507-3cc901e4ad3f" containerName="revoke-keys"
Sep 11 18:54:26.343663 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.343553    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="25edfbe6-858a-4c6c-8507-3cc901e4ad3f" containerName="revoke-keys"
Sep 11 18:54:26.343663 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.343612    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="25edfbe6-858a-4c6c-8507-3cc901e4ad3f" containerName="revoke-keys"
Sep 11 18:54:26.343663 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.343623    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e455243-6075-4137-ba60-a5b81dae2c3e" containerName="payload-processing"
Sep 11 18:54:26.343663 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.343630    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fea0428-cb51-4a39-b775-e0f8833b3e1b" containerName="payload-pre-processing"
Sep 11 18:54:26.348253 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.348232    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.350628 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.350606    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-722cbba0-kserve-self-signed-certs\""
Sep 11 18:54:26.355885 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.355863    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"]
Sep 11 18:54:26.495145 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.495095    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-home\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.495145 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.495153    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-tmp-dir\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.495379 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.495205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118a3381-612a-44fb-bfb3-5ba4aca085a3-tls-certs\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.495379 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.495286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-dshm\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.495379 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.495309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-model-cache\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.495379 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.495327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glg5l\" (UniqueName: \"kubernetes.io/projected/118a3381-612a-44fb-bfb3-5ba4aca085a3-kube-api-access-glg5l\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.595994 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.595905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-home\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.595994 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.595953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-tmp-dir\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.595994 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.595984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118a3381-612a-44fb-bfb3-5ba4aca085a3-tls-certs\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.596280 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.596014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-dshm\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.596280 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.596044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-model-cache\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.596280 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.596072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glg5l\" (UniqueName: \"kubernetes.io/projected/118a3381-612a-44fb-bfb3-5ba4aca085a3-kube-api-access-glg5l\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.596448 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.596376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-tmp-dir\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.596561 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.596444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-model-cache\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.596561 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.596553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-home\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.598363 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.598341    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-dshm\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.598642 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.598622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118a3381-612a-44fb-bfb3-5ba4aca085a3-tls-certs\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.605650 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.605626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glg5l\" (UniqueName: \"kubernetes.io/projected/118a3381-612a-44fb-bfb3-5ba4aca085a3-kube-api-access-glg5l\") pod \"e2e-same-id-a-722cbba0-kserve-666798895-xpb6m\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") " pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.658735 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.658689    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:26.934959 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.934916    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"]
Sep 11 18:54:26.937550 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:54:26.937515    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod118a3381_612a_44fb_bfb3_5ba4aca085a3.slice/crio-5635d9ec45205be1f4477d68a2c9d6b3ed703ed3680b020bd63fac129c32a68b WatchSource:0}: Error finding container 5635d9ec45205be1f4477d68a2c9d6b3ed703ed3680b020bd63fac129c32a68b: Status 404 returned error can't find the container with id 5635d9ec45205be1f4477d68a2c9d6b3ed703ed3680b020bd63fac129c32a68b
Sep 11 18:54:26.962948 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:26.962920    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m" event={"ID":"118a3381-612a-44fb-bfb3-5ba4aca085a3","Type":"ContainerStarted","Data":"5635d9ec45205be1f4477d68a2c9d6b3ed703ed3680b020bd63fac129c32a68b"}
Sep 11 18:54:27.969107 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:27.969071    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m" event={"ID":"118a3381-612a-44fb-bfb3-5ba4aca085a3","Type":"ContainerStarted","Data":"7fb70e19ad8e8473cd9af242e6a2ff3a6362995df148f323059f2222c2030fbe"}
Sep 11 18:54:27.992395 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:27.992334    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m" podStartSLOduration=1.992317672 podStartE2EDuration="1.992317672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:54:27.99127617 +0000 UTC m=+2344.732403014" watchObservedRunningTime="2026-09-11 18:54:27.992317672 +0000 UTC m=+2344.733444514"
Sep 11 18:54:28.856040 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:28.855998    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:54:36.658994 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:36.658950    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:36.659379 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:36.659011    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:36.672276 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:36.672245    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:36.940822 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:36.940698    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"]
Sep 11 18:54:36.944549 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:36.944525    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:36.946527 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:36.946502    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-722cbba0-kserve-self-signed-certs\""
Sep 11 18:54:36.952176 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:36.952146    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"]
Sep 11 18:54:37.031282 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.031239    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:54:37.098804 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.098769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-dshm\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.098991 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.098823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-home\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.098991 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.098855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-model-cache\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.098991 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.098948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrpcf\" (UniqueName: \"kubernetes.io/projected/f7eaaef4-205a-4f12-85a9-883f74aa5fce-kube-api-access-qrpcf\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.099166 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.099026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-tmp-dir\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.099166 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.099075    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7eaaef4-205a-4f12-85a9-883f74aa5fce-tls-certs\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.200352 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.200256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7eaaef4-205a-4f12-85a9-883f74aa5fce-tls-certs\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.200352 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.200319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-dshm\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.200352 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.200347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-home\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.200691 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.200374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-model-cache\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.200691 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.200406    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrpcf\" (UniqueName: \"kubernetes.io/projected/f7eaaef4-205a-4f12-85a9-883f74aa5fce-kube-api-access-qrpcf\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.200691 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.200613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-tmp-dir\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.200945 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.200907    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-home\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.201069 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.200945    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-model-cache\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.201069 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.200969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-tmp-dir\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.202865 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.202843    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-dshm\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.203117 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.203099    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7eaaef4-205a-4f12-85a9-883f74aa5fce-tls-certs\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.209236 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.209210    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrpcf\" (UniqueName: \"kubernetes.io/projected/f7eaaef4-205a-4f12-85a9-883f74aa5fce-kube-api-access-qrpcf\") pod \"e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") " pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.256675 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.256631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:37.529881 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:37.529772    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"]
Sep 11 18:54:37.532625 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:54:37.532595    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7eaaef4_205a_4f12_85a9_883f74aa5fce.slice/crio-54ad0b1fd28091d3991eaa2ffbb89ef469622ed89de5e4fe81851147e8e47518 WatchSource:0}: Error finding container 54ad0b1fd28091d3991eaa2ffbb89ef469622ed89de5e4fe81851147e8e47518: Status 404 returned error can't find the container with id 54ad0b1fd28091d3991eaa2ffbb89ef469622ed89de5e4fe81851147e8e47518
Sep 11 18:54:38.026537 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:38.026498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8" event={"ID":"f7eaaef4-205a-4f12-85a9-883f74aa5fce","Type":"ContainerStarted","Data":"0ed71120b3e1b388cae218fcad97eaa85dc47d1f6cc569b4e9654730bc67b2a4"}
Sep 11 18:54:38.026537 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:38.026538    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8" event={"ID":"f7eaaef4-205a-4f12-85a9-883f74aa5fce","Type":"ContainerStarted","Data":"54ad0b1fd28091d3991eaa2ffbb89ef469622ed89de5e4fe81851147e8e47518"}
Sep 11 18:54:38.043790 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:38.043728    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8" podStartSLOduration=2.043710955 podStartE2EDuration="2.043710955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:54:38.041671876 +0000 UTC m=+2354.782798720" watchObservedRunningTime="2026-09-11 18:54:38.043710955 +0000 UTC m=+2354.784837834"
Sep 11 18:54:39.752543 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:39.752505    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:54:47.257374 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:47.257322    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:47.257913 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:47.257399    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:47.270338 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:47.270311    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:48.087645 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:48.087612    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:54:53.360364 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:53.360330    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"]
Sep 11 18:54:54.759415 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:54.759377    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"]
Sep 11 18:54:55.764469 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:54:55.764434    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:55:04.940120 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:04.940082    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:55:08.380424 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.380379    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m" podUID="118a3381-612a-44fb-bfb3-5ba4aca085a3" containerName="main" containerID="cri-o://7fb70e19ad8e8473cd9af242e6a2ff3a6362995df148f323059f2222c2030fbe" gracePeriod=45
Sep 11 18:55:08.626524 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.626495    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:55:08.687820 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.687721    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-tmp-dir\") pod \"118a3381-612a-44fb-bfb3-5ba4aca085a3\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") "
Sep 11 18:55:08.687820 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.687782    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118a3381-612a-44fb-bfb3-5ba4aca085a3-tls-certs\") pod \"118a3381-612a-44fb-bfb3-5ba4aca085a3\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") "
Sep 11 18:55:08.687820 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.687820    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-model-cache\") pod \"118a3381-612a-44fb-bfb3-5ba4aca085a3\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") "
Sep 11 18:55:08.688071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.687847    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-home\") pod \"118a3381-612a-44fb-bfb3-5ba4aca085a3\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") "
Sep 11 18:55:08.688071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.687886    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glg5l\" (UniqueName: \"kubernetes.io/projected/118a3381-612a-44fb-bfb3-5ba4aca085a3-kube-api-access-glg5l\") pod \"118a3381-612a-44fb-bfb3-5ba4aca085a3\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") "
Sep 11 18:55:08.688071 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.687952    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-dshm\") pod \"118a3381-612a-44fb-bfb3-5ba4aca085a3\" (UID: \"118a3381-612a-44fb-bfb3-5ba4aca085a3\") "
Sep 11 18:55:08.688221 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.688059    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "118a3381-612a-44fb-bfb3-5ba4aca085a3" (UID: "118a3381-612a-44fb-bfb3-5ba4aca085a3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:55:08.688221 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.688067    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-model-cache" (OuterVolumeSpecName: "model-cache") pod "118a3381-612a-44fb-bfb3-5ba4aca085a3" (UID: "118a3381-612a-44fb-bfb3-5ba4aca085a3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:55:08.688221 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.688193    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-home" (OuterVolumeSpecName: "home") pod "118a3381-612a-44fb-bfb3-5ba4aca085a3" (UID: "118a3381-612a-44fb-bfb3-5ba4aca085a3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:55:08.688335 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.688315    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:55:08.688373 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.688334    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:55:08.688373 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.688354    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:55:08.690292 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.690259    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/118a3381-612a-44fb-bfb3-5ba4aca085a3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "118a3381-612a-44fb-bfb3-5ba4aca085a3" (UID: "118a3381-612a-44fb-bfb3-5ba4aca085a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:55:08.690292 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.690270    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-dshm" (OuterVolumeSpecName: "dshm") pod "118a3381-612a-44fb-bfb3-5ba4aca085a3" (UID: "118a3381-612a-44fb-bfb3-5ba4aca085a3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:55:08.690443 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.690310    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118a3381-612a-44fb-bfb3-5ba4aca085a3-kube-api-access-glg5l" (OuterVolumeSpecName: "kube-api-access-glg5l") pod "118a3381-612a-44fb-bfb3-5ba4aca085a3" (UID: "118a3381-612a-44fb-bfb3-5ba4aca085a3"). InnerVolumeSpecName "kube-api-access-glg5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:55:08.789072 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.789030    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/118a3381-612a-44fb-bfb3-5ba4aca085a3-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:55:08.789072 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.789064    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glg5l\" (UniqueName: \"kubernetes.io/projected/118a3381-612a-44fb-bfb3-5ba4aca085a3-kube-api-access-glg5l\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:55:08.789072 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:08.789074    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/118a3381-612a-44fb-bfb3-5ba4aca085a3-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:55:09.182710 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:09.182670    2581 generic.go:356] "Generic (PLEG): container finished" podID="118a3381-612a-44fb-bfb3-5ba4aca085a3" containerID="7fb70e19ad8e8473cd9af242e6a2ff3a6362995df148f323059f2222c2030fbe" exitCode=0
Sep 11 18:55:09.182893 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:09.182729    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"
Sep 11 18:55:09.182893 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:09.182738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m" event={"ID":"118a3381-612a-44fb-bfb3-5ba4aca085a3","Type":"ContainerDied","Data":"7fb70e19ad8e8473cd9af242e6a2ff3a6362995df148f323059f2222c2030fbe"}
Sep 11 18:55:09.182893 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:09.182778    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m" event={"ID":"118a3381-612a-44fb-bfb3-5ba4aca085a3","Type":"ContainerDied","Data":"5635d9ec45205be1f4477d68a2c9d6b3ed703ed3680b020bd63fac129c32a68b"}
Sep 11 18:55:09.182893 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:09.182799    2581 scope.go:117] "RemoveContainer" containerID="7fb70e19ad8e8473cd9af242e6a2ff3a6362995df148f323059f2222c2030fbe"
Sep 11 18:55:09.192314 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:09.192292    2581 scope.go:117] "RemoveContainer" containerID="7fb70e19ad8e8473cd9af242e6a2ff3a6362995df148f323059f2222c2030fbe"
Sep 11 18:55:09.192659 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:55:09.192636    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb70e19ad8e8473cd9af242e6a2ff3a6362995df148f323059f2222c2030fbe\": container with ID starting with 7fb70e19ad8e8473cd9af242e6a2ff3a6362995df148f323059f2222c2030fbe not found: ID does not exist" containerID="7fb70e19ad8e8473cd9af242e6a2ff3a6362995df148f323059f2222c2030fbe"
Sep 11 18:55:09.192740 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:09.192672    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb70e19ad8e8473cd9af242e6a2ff3a6362995df148f323059f2222c2030fbe"} err="failed to get container status \"7fb70e19ad8e8473cd9af242e6a2ff3a6362995df148f323059f2222c2030fbe\": rpc error: code = NotFound desc = could not find container \"7fb70e19ad8e8473cd9af242e6a2ff3a6362995df148f323059f2222c2030fbe\": container with ID starting with 7fb70e19ad8e8473cd9af242e6a2ff3a6362995df148f323059f2222c2030fbe not found: ID does not exist"
Sep 11 18:55:09.204199 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:09.204167    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"]
Sep 11 18:55:09.208893 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:09.208866    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-722cbba0-kserve-666798895-xpb6m"]
Sep 11 18:55:09.763559 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:09.763524    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="118a3381-612a-44fb-bfb3-5ba4aca085a3" path="/var/lib/kubelet/pods/118a3381-612a-44fb-bfb3-5ba4aca085a3/volumes"
Sep 11 18:55:09.779819 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:09.779781    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8" podUID="f7eaaef4-205a-4f12-85a9-883f74aa5fce" containerName="main" containerID="cri-o://0ed71120b3e1b388cae218fcad97eaa85dc47d1f6cc569b4e9654730bc67b2a4" gracePeriod=45
Sep 11 18:55:10.025002 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.024938    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:55:10.100049 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.100008    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrpcf\" (UniqueName: \"kubernetes.io/projected/f7eaaef4-205a-4f12-85a9-883f74aa5fce-kube-api-access-qrpcf\") pod \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") "
Sep 11 18:55:10.100230 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.100094    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-home\") pod \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") "
Sep 11 18:55:10.100276 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.100251    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-model-cache\") pod \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") "
Sep 11 18:55:10.100311 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.100286    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7eaaef4-205a-4f12-85a9-883f74aa5fce-tls-certs\") pod \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") "
Sep 11 18:55:10.100311 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.100305    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-tmp-dir\") pod \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") "
Sep 11 18:55:10.100413 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.100309    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-home" (OuterVolumeSpecName: "home") pod "f7eaaef4-205a-4f12-85a9-883f74aa5fce" (UID: "f7eaaef4-205a-4f12-85a9-883f74aa5fce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:55:10.100413 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.100331    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-dshm\") pod \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\" (UID: \"f7eaaef4-205a-4f12-85a9-883f74aa5fce\") "
Sep 11 18:55:10.100572 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.100527    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7eaaef4-205a-4f12-85a9-883f74aa5fce" (UID: "f7eaaef4-205a-4f12-85a9-883f74aa5fce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:55:10.100572 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.100544    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7eaaef4-205a-4f12-85a9-883f74aa5fce" (UID: "f7eaaef4-205a-4f12-85a9-883f74aa5fce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:55:10.100743 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.100723    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-home\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:55:10.100816 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.100743    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-model-cache\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:55:10.100816 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.100753    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-tmp-dir\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:55:10.102359 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.102332    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7eaaef4-205a-4f12-85a9-883f74aa5fce-kube-api-access-qrpcf" (OuterVolumeSpecName: "kube-api-access-qrpcf") pod "f7eaaef4-205a-4f12-85a9-883f74aa5fce" (UID: "f7eaaef4-205a-4f12-85a9-883f74aa5fce"). InnerVolumeSpecName "kube-api-access-qrpcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:55:10.102463 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.102442    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7eaaef4-205a-4f12-85a9-883f74aa5fce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7eaaef4-205a-4f12-85a9-883f74aa5fce" (UID: "f7eaaef4-205a-4f12-85a9-883f74aa5fce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:55:10.102591 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.102507    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-dshm" (OuterVolumeSpecName: "dshm") pod "f7eaaef4-205a-4f12-85a9-883f74aa5fce" (UID: "f7eaaef4-205a-4f12-85a9-883f74aa5fce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:55:10.191702 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.191669    2581 generic.go:356] "Generic (PLEG): container finished" podID="f7eaaef4-205a-4f12-85a9-883f74aa5fce" containerID="0ed71120b3e1b388cae218fcad97eaa85dc47d1f6cc569b4e9654730bc67b2a4" exitCode=0
Sep 11 18:55:10.191885 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.191713    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"
Sep 11 18:55:10.191885 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.191733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8" event={"ID":"f7eaaef4-205a-4f12-85a9-883f74aa5fce","Type":"ContainerDied","Data":"0ed71120b3e1b388cae218fcad97eaa85dc47d1f6cc569b4e9654730bc67b2a4"}
Sep 11 18:55:10.191885 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.191762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8" event={"ID":"f7eaaef4-205a-4f12-85a9-883f74aa5fce","Type":"ContainerDied","Data":"54ad0b1fd28091d3991eaa2ffbb89ef469622ed89de5e4fe81851147e8e47518"}
Sep 11 18:55:10.191885 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.191778    2581 scope.go:117] "RemoveContainer" containerID="0ed71120b3e1b388cae218fcad97eaa85dc47d1f6cc569b4e9654730bc67b2a4"
Sep 11 18:55:10.201410 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.201333    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7eaaef4-205a-4f12-85a9-883f74aa5fce-tls-certs\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:55:10.201410 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.201363    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7eaaef4-205a-4f12-85a9-883f74aa5fce-dshm\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:55:10.201410 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.201376    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrpcf\" (UniqueName: \"kubernetes.io/projected/f7eaaef4-205a-4f12-85a9-883f74aa5fce-kube-api-access-qrpcf\") on node \"ip-10-0-129-166.ec2.internal\" DevicePath \"\""
Sep 11 18:55:10.205300 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.205276    2581 scope.go:117] "RemoveContainer" containerID="0ed71120b3e1b388cae218fcad97eaa85dc47d1f6cc569b4e9654730bc67b2a4"
Sep 11 18:55:10.205691 ip-10-0-129-166 kubenswrapper[2581]: E0911 18:55:10.205665    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed71120b3e1b388cae218fcad97eaa85dc47d1f6cc569b4e9654730bc67b2a4\": container with ID starting with 0ed71120b3e1b388cae218fcad97eaa85dc47d1f6cc569b4e9654730bc67b2a4 not found: ID does not exist" containerID="0ed71120b3e1b388cae218fcad97eaa85dc47d1f6cc569b4e9654730bc67b2a4"
Sep 11 18:55:10.205807 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.205699    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed71120b3e1b388cae218fcad97eaa85dc47d1f6cc569b4e9654730bc67b2a4"} err="failed to get container status \"0ed71120b3e1b388cae218fcad97eaa85dc47d1f6cc569b4e9654730bc67b2a4\": rpc error: code = NotFound desc = could not find container \"0ed71120b3e1b388cae218fcad97eaa85dc47d1f6cc569b4e9654730bc67b2a4\": container with ID starting with 0ed71120b3e1b388cae218fcad97eaa85dc47d1f6cc569b4e9654730bc67b2a4 not found: ID does not exist"
Sep 11 18:55:10.215437 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.215408    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"]
Sep 11 18:55:10.220106 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:10.220076    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-722cbba0-kserve-6dc894ff55-k65k8"]
Sep 11 18:55:11.765218 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:11.765184    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7eaaef4-205a-4f12-85a9-883f74aa5fce" path="/var/lib/kubelet/pods/f7eaaef4-205a-4f12-85a9-883f74aa5fce/volumes"
Sep 11 18:55:12.860803 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:12.860764    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:55:20.243885 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:20.243848    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:55:24.292888 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:24.292850    2581 scope.go:117] "RemoveContainer" containerID="48624489da40737a771dc362c14403ea43cfb967a49aaafb0ad71ebec38ca2b6"
Sep 11 18:55:27.450211 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:27.450173    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:55:35.947212 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:35.947177    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:55:43.548748 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:43.548709    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-mkgcd"]
Sep 11 18:55:52.518118 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:52.518036    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cff9b6b6-lm8nq_7059e104-48c1-4026-b701-eceb3e64f340/manager/0.log"
Sep 11 18:55:52.525141 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:52.525115    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cff9b6b6-lm8nq_7059e104-48c1-4026-b701-eceb3e64f340/copy-manifests/0.log"
Sep 11 18:55:53.439997 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:53.439963    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819205-p5rcm_154a5d37-e072-4f4c-bf3d-4f5c8253d753/cleanup/0.log"
Sep 11 18:55:54.531396 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:54.531366    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-56gf4_b2ba9e8b-1e6f-4a80-8b0c-39f09e1fb8be/manager/0.log"
Sep 11 18:55:54.648410 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:54.648381    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-n6nh7_edaa18e3-d95c-40f2-84d7-7271b9921e11/kuadrant-console-plugin/0.log"
Sep 11 18:55:54.884394 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:54.884368    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-mkgcd_edff1179-2a27-4a36-9373-46cab7d17a9f/limitador/0.log"
Sep 11 18:55:55.590763 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:55.590729    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67fd44999f-9nfjj_917d39f0-df45-4535-8f45-273b74114c36/kube-auth-proxy/0.log"
Sep 11 18:55:56.071310 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:56.071280    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2292w_cc357e4e-df13-4296-84eb-8d0d846b34be/payload-processing/0.log"
Sep 11 18:55:56.180701 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:56.180669    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7dcc55bd66-526gs_c230cd27-7b95-48a6-8cd8-9f10f8a87d8a/router/0.log"
Sep 11 18:55:56.852620 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:56.852587    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-f7mc2_3d9faf35-7782-4f47-8924-13f951123424/main/0.log"
Sep 11 18:55:57.185195 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:55:57.185159    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-w8w48_754e8a1e-d3bf-4d20-b0d6-3ae61a880424/main/0.log"
Sep 11 18:56:02.261950 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.261908    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kfqdv/must-gather-rz7b6"]
Sep 11 18:56:02.262334 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.262280    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7eaaef4-205a-4f12-85a9-883f74aa5fce" containerName="main"
Sep 11 18:56:02.262334 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.262292    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7eaaef4-205a-4f12-85a9-883f74aa5fce" containerName="main"
Sep 11 18:56:02.262334 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.262321    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="118a3381-612a-44fb-bfb3-5ba4aca085a3" containerName="main"
Sep 11 18:56:02.262334 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.262326    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="118a3381-612a-44fb-bfb3-5ba4aca085a3" containerName="main"
Sep 11 18:56:02.262471 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.262381    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="118a3381-612a-44fb-bfb3-5ba4aca085a3" containerName="main"
Sep 11 18:56:02.262471 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.262390    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7eaaef4-205a-4f12-85a9-883f74aa5fce" containerName="main"
Sep 11 18:56:02.265519 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.265501    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfqdv/must-gather-rz7b6"
Sep 11 18:56:02.267683 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.267659    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kfqdv\"/\"openshift-service-ca.crt\""
Sep 11 18:56:02.267683 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.267660    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kfqdv\"/\"kube-root-ca.crt\""
Sep 11 18:56:02.277776 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.277747    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfqdv/must-gather-rz7b6"]
Sep 11 18:56:02.379072 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.379025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/477311fb-2246-437a-bb05-bb965e5b9386-must-gather-output\") pod \"must-gather-rz7b6\" (UID: \"477311fb-2246-437a-bb05-bb965e5b9386\") " pod="openshift-must-gather-kfqdv/must-gather-rz7b6"
Sep 11 18:56:02.379072 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.379081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzqrb\" (UniqueName: \"kubernetes.io/projected/477311fb-2246-437a-bb05-bb965e5b9386-kube-api-access-lzqrb\") pod \"must-gather-rz7b6\" (UID: \"477311fb-2246-437a-bb05-bb965e5b9386\") " pod="openshift-must-gather-kfqdv/must-gather-rz7b6"
Sep 11 18:56:02.480380 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.480343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/477311fb-2246-437a-bb05-bb965e5b9386-must-gather-output\") pod \"must-gather-rz7b6\" (UID: \"477311fb-2246-437a-bb05-bb965e5b9386\") " pod="openshift-must-gather-kfqdv/must-gather-rz7b6"
Sep 11 18:56:02.480596 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.480389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzqrb\" (UniqueName: \"kubernetes.io/projected/477311fb-2246-437a-bb05-bb965e5b9386-kube-api-access-lzqrb\") pod \"must-gather-rz7b6\" (UID: \"477311fb-2246-437a-bb05-bb965e5b9386\") " pod="openshift-must-gather-kfqdv/must-gather-rz7b6"
Sep 11 18:56:02.480749 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.480721    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/477311fb-2246-437a-bb05-bb965e5b9386-must-gather-output\") pod \"must-gather-rz7b6\" (UID: \"477311fb-2246-437a-bb05-bb965e5b9386\") " pod="openshift-must-gather-kfqdv/must-gather-rz7b6"
Sep 11 18:56:02.489717 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.489692    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzqrb\" (UniqueName: \"kubernetes.io/projected/477311fb-2246-437a-bb05-bb965e5b9386-kube-api-access-lzqrb\") pod \"must-gather-rz7b6\" (UID: \"477311fb-2246-437a-bb05-bb965e5b9386\") " pod="openshift-must-gather-kfqdv/must-gather-rz7b6"
Sep 11 18:56:02.576759 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.576669    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfqdv/must-gather-rz7b6"
Sep 11 18:56:02.856750 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.856722    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfqdv/must-gather-rz7b6"]
Sep 11 18:56:02.859019 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:56:02.858991    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod477311fb_2246_437a_bb05_bb965e5b9386.slice/crio-45c9c7aa711e6bfefb74157be4f3c0a533b9dd144d3b8ecbbf2388337a5ca55c WatchSource:0}: Error finding container 45c9c7aa711e6bfefb74157be4f3c0a533b9dd144d3b8ecbbf2388337a5ca55c: Status 404 returned error can't find the container with id 45c9c7aa711e6bfefb74157be4f3c0a533b9dd144d3b8ecbbf2388337a5ca55c
Sep 11 18:56:02.860829 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:02.860807    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:56:03.456711 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:03.456672    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfqdv/must-gather-rz7b6" event={"ID":"477311fb-2246-437a-bb05-bb965e5b9386","Type":"ContainerStarted","Data":"45c9c7aa711e6bfefb74157be4f3c0a533b9dd144d3b8ecbbf2388337a5ca55c"}
Sep 11 18:56:04.466817 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:04.466769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfqdv/must-gather-rz7b6" event={"ID":"477311fb-2246-437a-bb05-bb965e5b9386","Type":"ContainerStarted","Data":"94c2e5a58b3d6b56b52bdde224c58e30582095ac65f7e3df96ccaf77ae9d96a9"}
Sep 11 18:56:04.467509 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:04.467443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfqdv/must-gather-rz7b6" event={"ID":"477311fb-2246-437a-bb05-bb965e5b9386","Type":"ContainerStarted","Data":"ddc1e4e35089d689838be5b8efe0bf72bcc5a50f201601a06d06da18ff3374ab"}
Sep 11 18:56:05.399572 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:05.399533    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9rs25_eb4f3610-8190-4056-88be-96c5087cb949/global-pull-secret-syncer/0.log"
Sep 11 18:56:05.578953 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:05.578918    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zdcg9_a0a651f9-e86d-459b-b079-d9a009db6c7c/konnectivity-agent/0.log"
Sep 11 18:56:05.618152 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:05.618125    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-166.ec2.internal_1fccd7d8530b9bbb746ecca8d0ec2393/haproxy/0.log"
Sep 11 18:56:10.659398 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:10.659357    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-56gf4_b2ba9e8b-1e6f-4a80-8b0c-39f09e1fb8be/manager/0.log"
Sep 11 18:56:10.708219 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:10.708160    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-n6nh7_edaa18e3-d95c-40f2-84d7-7271b9921e11/kuadrant-console-plugin/0.log"
Sep 11 18:56:10.783596 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:10.783547    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-mkgcd_edff1179-2a27-4a36-9373-46cab7d17a9f/limitador/0.log"
Sep 11 18:56:12.418897 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:12.418839    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df23b4f0-1ab0-4450-96e5-aba957a0c89e/alertmanager/0.log"
Sep 11 18:56:12.447081 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:12.447051    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df23b4f0-1ab0-4450-96e5-aba957a0c89e/config-reloader/0.log"
Sep 11 18:56:12.470995 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:12.470964    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df23b4f0-1ab0-4450-96e5-aba957a0c89e/kube-rbac-proxy-web/0.log"
Sep 11 18:56:12.499690 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:12.499657    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df23b4f0-1ab0-4450-96e5-aba957a0c89e/kube-rbac-proxy/0.log"
Sep 11 18:56:12.530553 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:12.530395    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df23b4f0-1ab0-4450-96e5-aba957a0c89e/kube-rbac-proxy-metric/0.log"
Sep 11 18:56:12.554332 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:12.554284    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df23b4f0-1ab0-4450-96e5-aba957a0c89e/prom-label-proxy/0.log"
Sep 11 18:56:12.582087 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:12.582053    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_df23b4f0-1ab0-4450-96e5-aba957a0c89e/init-config-reloader/0.log"
Sep 11 18:56:12.753457 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:12.753431    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b4fbbccdd-rtlqf_57c68838-049b-41e9-957c-a00e44516563/metrics-server/0.log"
Sep 11 18:56:12.796718 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:12.796587    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-bm4mc_91949a49-5e56-4b47-bdaa-1900732d79e3/monitoring-plugin/0.log"
Sep 11 18:56:13.043642 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.043608    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l7n9c_c332143e-9818-402b-9020-b419b5ad7f74/node-exporter/0.log"
Sep 11 18:56:13.070013 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.069929    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l7n9c_c332143e-9818-402b-9020-b419b5ad7f74/kube-rbac-proxy/0.log"
Sep 11 18:56:13.094461 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.094430    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l7n9c_c332143e-9818-402b-9020-b419b5ad7f74/init-textfile/0.log"
Sep 11 18:56:13.141003 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.140966    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-n7wxx_1079d587-e3dc-471f-8cdb-11d28d7ae5d1/kube-rbac-proxy-main/0.log"
Sep 11 18:56:13.165329 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.165300    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-n7wxx_1079d587-e3dc-471f-8cdb-11d28d7ae5d1/kube-rbac-proxy-self/0.log"
Sep 11 18:56:13.191621 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.191592    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-n7wxx_1079d587-e3dc-471f-8cdb-11d28d7ae5d1/openshift-state-metrics/0.log"
Sep 11 18:56:13.237039 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.237006    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_be126077-8ac6-4292-a6fd-19c933373f63/prometheus/0.log"
Sep 11 18:56:13.261187 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.261135    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_be126077-8ac6-4292-a6fd-19c933373f63/config-reloader/0.log"
Sep 11 18:56:13.285522 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.285488    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_be126077-8ac6-4292-a6fd-19c933373f63/thanos-sidecar/0.log"
Sep 11 18:56:13.309611 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.309530    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_be126077-8ac6-4292-a6fd-19c933373f63/kube-rbac-proxy-web/0.log"
Sep 11 18:56:13.335717 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.335639    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_be126077-8ac6-4292-a6fd-19c933373f63/kube-rbac-proxy/0.log"
Sep 11 18:56:13.362756 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.362726    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_be126077-8ac6-4292-a6fd-19c933373f63/kube-rbac-proxy-thanos/0.log"
Sep 11 18:56:13.396722 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.396696    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_be126077-8ac6-4292-a6fd-19c933373f63/init-config-reloader/0.log"
Sep 11 18:56:13.434051 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.434015    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-hkj9k_ac54b5e3-0538-4542-8848-97249c328418/prometheus-operator/0.log"
Sep 11 18:56:13.458270 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.458236    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-hkj9k_ac54b5e3-0538-4542-8848-97249c328418/kube-rbac-proxy/0.log"
Sep 11 18:56:13.486135 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.486097    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-j2gmd_705fef02-989f-44dd-b52b-9283d167e1f8/prometheus-operator-admission-webhook/0.log"
Sep 11 18:56:13.521673 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.521635    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54d8895f84-wkfkb_0dc77db2-d8b5-49f3-8894-f5a193c9fb08/telemeter-client/0.log"
Sep 11 18:56:13.549369 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.549337    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54d8895f84-wkfkb_0dc77db2-d8b5-49f3-8894-f5a193c9fb08/reload/0.log"
Sep 11 18:56:13.574724 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:13.574696    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-54d8895f84-wkfkb_0dc77db2-d8b5-49f3-8894-f5a193c9fb08/kube-rbac-proxy/0.log"
Sep 11 18:56:15.580322 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.580268    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kfqdv/must-gather-rz7b6" podStartSLOduration=12.814984565 podStartE2EDuration="13.580245382s" totalImagesPullingTime="765.260817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:56:04.487173395 +0000 UTC m=+2441.228300238" watchObservedRunningTime="2026-09-11 18:56:15.580245382 +0000 UTC m=+2452.321372229"
Sep 11 18:56:15.580848 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.580772    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"]
Sep 11 18:56:15.586889 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.586862    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:15.589936 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.589906    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kfqdv\"/\"default-dockercfg-kn4ch\""
Sep 11 18:56:15.593297 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.593269    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"]
Sep 11 18:56:15.732720 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.732670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5df49e48-6468-45d9-b356-37806f5bee33-proc\") pod \"perf-node-gather-daemonset-bx76r\" (UID: \"5df49e48-6468-45d9-b356-37806f5bee33\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:15.732925 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.732786    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5df49e48-6468-45d9-b356-37806f5bee33-sys\") pod \"perf-node-gather-daemonset-bx76r\" (UID: \"5df49e48-6468-45d9-b356-37806f5bee33\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:15.732925 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.732901    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5df49e48-6468-45d9-b356-37806f5bee33-lib-modules\") pod \"perf-node-gather-daemonset-bx76r\" (UID: \"5df49e48-6468-45d9-b356-37806f5bee33\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:15.733061 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.732970    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8vt4\" (UniqueName: \"kubernetes.io/projected/5df49e48-6468-45d9-b356-37806f5bee33-kube-api-access-c8vt4\") pod \"perf-node-gather-daemonset-bx76r\" (UID: \"5df49e48-6468-45d9-b356-37806f5bee33\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:15.733061 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.733005    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5df49e48-6468-45d9-b356-37806f5bee33-podres\") pod \"perf-node-gather-daemonset-bx76r\" (UID: \"5df49e48-6468-45d9-b356-37806f5bee33\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:15.834522 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.834404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5df49e48-6468-45d9-b356-37806f5bee33-sys\") pod \"perf-node-gather-daemonset-bx76r\" (UID: \"5df49e48-6468-45d9-b356-37806f5bee33\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:15.834840 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.834813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5df49e48-6468-45d9-b356-37806f5bee33-lib-modules\") pod \"perf-node-gather-daemonset-bx76r\" (UID: \"5df49e48-6468-45d9-b356-37806f5bee33\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:15.835164 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.835142    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8vt4\" (UniqueName: \"kubernetes.io/projected/5df49e48-6468-45d9-b356-37806f5bee33-kube-api-access-c8vt4\") pod \"perf-node-gather-daemonset-bx76r\" (UID: \"5df49e48-6468-45d9-b356-37806f5bee33\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:15.835545 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.835520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5df49e48-6468-45d9-b356-37806f5bee33-podres\") pod \"perf-node-gather-daemonset-bx76r\" (UID: \"5df49e48-6468-45d9-b356-37806f5bee33\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:15.835665 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.835087    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5df49e48-6468-45d9-b356-37806f5bee33-lib-modules\") pod \"perf-node-gather-daemonset-bx76r\" (UID: \"5df49e48-6468-45d9-b356-37806f5bee33\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:15.835665 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.834640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5df49e48-6468-45d9-b356-37806f5bee33-sys\") pod \"perf-node-gather-daemonset-bx76r\" (UID: \"5df49e48-6468-45d9-b356-37806f5bee33\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:15.835783 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.835694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5df49e48-6468-45d9-b356-37806f5bee33-proc\") pod \"perf-node-gather-daemonset-bx76r\" (UID: \"5df49e48-6468-45d9-b356-37806f5bee33\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:15.835783 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.835702    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5df49e48-6468-45d9-b356-37806f5bee33-podres\") pod \"perf-node-gather-daemonset-bx76r\" (UID: \"5df49e48-6468-45d9-b356-37806f5bee33\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:15.835783 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.835773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5df49e48-6468-45d9-b356-37806f5bee33-proc\") pod \"perf-node-gather-daemonset-bx76r\" (UID: \"5df49e48-6468-45d9-b356-37806f5bee33\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:15.845798 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.845770    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8vt4\" (UniqueName: \"kubernetes.io/projected/5df49e48-6468-45d9-b356-37806f5bee33-kube-api-access-c8vt4\") pod \"perf-node-gather-daemonset-bx76r\" (UID: \"5df49e48-6468-45d9-b356-37806f5bee33\") " pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:15.904532 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:15.904467    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:16.193207 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:16.193094    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"]
Sep 11 18:56:16.196298 ip-10-0-129-166 kubenswrapper[2581]: W0911 18:56:16.196262    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5df49e48_6468_45d9_b356_37806f5bee33.slice/crio-3f350424ee658d9b0b35f18c72823428aca563a73af837fcdb68eb9f762c0564 WatchSource:0}: Error finding container 3f350424ee658d9b0b35f18c72823428aca563a73af837fcdb68eb9f762c0564: Status 404 returned error can't find the container with id 3f350424ee658d9b0b35f18c72823428aca563a73af837fcdb68eb9f762c0564
Sep 11 18:56:16.398525 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:16.398465    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-br52f_4e69db3f-e2e2-4726-80cc-3da6ca8c73ec/download-server/0.log"
Sep 11 18:56:16.557904 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:16.557807    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r" event={"ID":"5df49e48-6468-45d9-b356-37806f5bee33","Type":"ContainerStarted","Data":"4450a13e438720fd824bb75aafcfc787a8267c43493627d641041a178d93180c"}
Sep 11 18:56:16.557904 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:16.557884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r" event={"ID":"5df49e48-6468-45d9-b356-37806f5bee33","Type":"ContainerStarted","Data":"3f350424ee658d9b0b35f18c72823428aca563a73af837fcdb68eb9f762c0564"}
Sep 11 18:56:16.558818 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:16.558784    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:16.576124 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:16.576054    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r" podStartSLOduration=1.576028389 podStartE2EDuration="1.576028389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:56:16.574925503 +0000 UTC m=+2453.316052348" watchObservedRunningTime="2026-09-11 18:56:16.576028389 +0000 UTC m=+2453.317155234"
Sep 11 18:56:18.391394 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:18.391362    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lck25_e8a008ce-8cd5-4753-a3ad-7337832bf965/dns/0.log"
Sep 11 18:56:18.414690 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:18.414653    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lck25_e8a008ce-8cd5-4753-a3ad-7337832bf965/kube-rbac-proxy/0.log"
Sep 11 18:56:18.438319 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:18.438291    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b9dfv_68330d27-5afb-44d1-96a2-774104ee6c4c/dns-node-resolver/0.log"
Sep 11 18:56:19.135698 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:19.135657    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xvv2h_f9719c07-ac09-4ed0-a4da-515e4d30fe2c/node-ca/0.log"
Sep 11 18:56:20.575524 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:20.575472    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67fd44999f-9nfjj_917d39f0-df45-4535-8f45-273b74114c36/kube-auth-proxy/0.log"
Sep 11 18:56:20.729276 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:20.729228    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2292w_cc357e4e-df13-4296-84eb-8d0d846b34be/payload-processing/0.log"
Sep 11 18:56:20.757684 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:20.757651    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7dcc55bd66-526gs_c230cd27-7b95-48a6-8cd8-9f10f8a87d8a/router/0.log"
Sep 11 18:56:21.662579 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:21.662542    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gg6sb_a6eb04a4-7bb3-4a28-90e8-1e09e88a35ba/serve-healthcheck-canary/0.log"
Sep 11 18:56:22.239934 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:22.239898    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bv5j5_96e20d2c-23a6-4e42-a459-e680cddc8572/kube-rbac-proxy/0.log"
Sep 11 18:56:22.269085 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:22.269044    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bv5j5_96e20d2c-23a6-4e42-a459-e680cddc8572/exporter/0.log"
Sep 11 18:56:22.293107 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:22.293076    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bv5j5_96e20d2c-23a6-4e42-a459-e680cddc8572/extractor/0.log"
Sep 11 18:56:23.585377 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:23.585344    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kfqdv/perf-node-gather-daemonset-bx76r"
Sep 11 18:56:24.319862 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:24.319721    2581 scope.go:117] "RemoveContainer" containerID="f0dc1b94d14529ff98f3d94f9273d25d0ea702f1c7002969e832d9c95645f486"
Sep 11 18:56:24.337903 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:24.330453    2581 scope.go:117] "RemoveContainer" containerID="aa90ef568d285463a9c463579edb8d9f90d2580c85841167d9e5164ec2f7e7c7"
Sep 11 18:56:25.728179 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:25.728139    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cff9b6b6-lm8nq_7059e104-48c1-4026-b701-eceb3e64f340/manager/0.log"
Sep 11 18:56:25.754213 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:25.754186    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76cff9b6b6-lm8nq_7059e104-48c1-4026-b701-eceb3e64f340/copy-manifests/0.log"
Sep 11 18:56:27.099356 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:27.099323    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67fd44999f-9nfjj_917d39f0-df45-4535-8f45-273b74114c36/kube-auth-proxy/0.log"
Sep 11 18:56:27.157668 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:27.157622    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5588b8499b-6r6r7_0ed83167-16c8-417b-9f27-077eefaf338b/manager/0.log"
Sep 11 18:56:27.209149 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:27.209066    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2292w_cc357e4e-df13-4296-84eb-8d0d846b34be/payload-processing/0.log"
Sep 11 18:56:27.225514 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:27.225458    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7dcc55bd66-526gs_c230cd27-7b95-48a6-8cd8-9f10f8a87d8a/router/0.log"
Sep 11 18:56:29.222860 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:29.222830    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-67fd44999f-9nfjj_917d39f0-df45-4535-8f45-273b74114c36/kube-auth-proxy/0.log"
Sep 11 18:56:29.334506 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:29.334454    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-2292w_cc357e4e-df13-4296-84eb-8d0d846b34be/payload-processing/0.log"
Sep 11 18:56:29.354673 ip-10-0-129-166 kubenswrapper[2581]: I0911 18:56:29.354645    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7dcc55bd66-526gs_c230cd27-7b95-48a6-8cd8-9f10f8a87d8a/router/0.log"